acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Influence Learning for Urban Spatio-Temporal Forecasting: A Cross-City Foundation Model and Dataset

66 views

Abstract

Urban spatio-temporal foundation models learn reusable forecasting patterns by pretraining across cities, yet they rely mainly on single-modal spatiotemporal observational data. Multimodal information such as roads, remote sensing, weather, and events provides predictive evidence beyond these observations, but transferring the learned use of multimodal information across cities remains an open problem. A further unaddressed challenge is that modalities contain heterogeneous content, and individually beneficial modalities can interfere when used jointly. To this end, we propose MILU, a multimodal urban spatio-temporal foundation model that learns each modality's predictive influence rather than directly mixing modality content. MILU encodes each modality separately, retrieves historical spatio-temporal context conditioned on both the forecasting query and modality information, and composes the resulting influences in a shared influence space to modulate forecasting states under bounded scaling. MILU is jointly pretrained across cities by comparing forecasting errors with and without multimodal information, so that it learns how to use multimodal information under varying availability. This transfer also requires pretraining data with diverse modalities, ample observations, and broad geographic coverage, which existing urban foundation model resources do not yet provide. We therefore construct the CityModal Dataset, covering 87 regions across four continents with approximately 19.21 billion valid spatio-temporal data points, and associate urban observations with road, remote-sensing, weather, and event information. Experiments show that, without target training, MILU achieves the lowest MAE on 28 of 30 short-term and 27 of 30 long-term forecasting tasks, and achieves increasing gains as more modalities are used jointly. With one tenth of the target training data, it outperforms the strongest task-specific baseline trained on the full target set in aggregate performance. Controlled comparisons further show that multimodal utilization transfers to new scenarios, extends to modality combinations unseen during training, and improves with limited target-data adaptation.

Primary area:
learning on time series and dynamical systems
Keywords:
Spatio-temporal forecasting, Spatio-temporal foundation models, Multimodal spatio-temporal forecasting

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AX73OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MILU, a multimodal foundation model for urban spatio-temporal forecasting that addresses interference among heterogeneous modalities. The method introduces modality-conditioned historical retrieval, shared influence learning, and bounded feature modulation, together with a multimodal pretraining strategy designed to improve cross-city transferability. The authors also construct CityModal, a large-scale dataset covering 87 geographic regions and approximately 19.21 billion valid spatio-temporal data points. Extensive experiments demonstrate strong zero-shot forecasting performance, effective few-shot adaptation, and generalization to unseen modality combinations.

Critical strengths
  1. Well-motivated methodology and strong empirical performance.The proposed shared influence learning framework provides a meaningful alternative to conventional multimodal fusion by explicitly addressing interference among heterogeneous modalities. MILU achieves the best MAE on 28/30 short-term and 27/30 long-term zero-shot forecasting tasks. The extensive experiments, including few-shot adaptation and ablation studies, provide convincing evidence of its effectiveness and transferability.

  2. Valuable large-scale dataset contribution. CityModal integrates diverse multimodal information across 87 geographic regions, containing approximately 19.21 billion valid spatio-temporal data points. Its substantial scale, modality diversity, and geographic coverage provide a valuable resource for future research on multimodal urban spatio-temporal foundation models.

Critical weaknesses
  1. Differences in pretraining data across baselines. As described in Appendix C.2, different foundation-model baselines use different pretraining corpora or publicly released checkpoints. Therefore, the zero-shot performance improvements in Table 2 may reflect both methodological advantages and differences in pretraining data. A clearer discussion of this distinction would strengthen the experimental conclusions.
  2. Comparability of multimodal input conditions. Section 6.2 compares models using their individually supported input modalities. Since some baselines do not utilize the same multimodal information as MILU, part of the performance difference may arise from differences in input availability. Although the authors provide controlled ablation experiments, the interpretation of these comparisons could be further clarified.
  3. Limited fine-grained analysis of multimodal collaboration. Figures 1 and 6 demonstrate encouraging aggregate improvements when multiple modalities are combined. However, these aggregated results do not fully reveal whether the proposed method consistently avoids negative interactions across individual forecasting tasks. Additional task-level analysis of joint-use gains relative to the best individual modality would further support the central motivation.

If the authors can satisfactorily address these concerns, I would be willing to increase my overall rating.

Questions for authors
  1. Pretraining Data Comparability: We noticed in Appendix C.2 that several baseline models use publicly released pretrained weights or are pretrained on different datasets (e.g., UrbanFM on WorldST), rather than being pretrained on the same corpus as MILU. Consequently, there may be substantial differences in pretraining data scale, diversity, and geographic coverage between MILU and these baselines. These differences could potentially contribute to the performance gaps observed in the zero-shot experiments in Table 2, making it difficult to attribute the improvements solely to the proposed methodology. If possible, additional evidence under more comparable pretraining conditions would further strengthen the conclusions.

  2. Fairness of Multimodal Comparisons: Since different baselines support different input modalities, could the authors clarify how the fairness of multimodal comparisons is ensured? Are there additional controlled comparisons under matched input conditions, or could the authors further distinguish gains from architectural improvements and those from access to additional modalities?

  3. Consistency of Multimodal Collaboration: The aggregate results suggest that combining multiple modalities improves forecasting performance. However, does this improvement consistently hold across individual forecasting tasks? Could the authors provide a more detailed analysis of joint-use gains relative to the best-performing individual modality, particularly highlighting cases where multimodal combination fails to provide additional benefits?

4 comments

Response to Reviewer AX73 — Part 1/2

Thanks for these valuable comments. We have conducted additional controlled experiments and task-level analyses to better clarify the sources of MILU's performance gains, ensure fairer comparisons across different input conditions, and examine the consistency of multimodal collaboration. We have also clarified the scope and interpretation of the corresponding experimental results in the revised manuscript.

W1 + Q1 — Pretraining data comparability

To address this concern, we have conducted additional controlled experiments by pretraining UrbanFM and OpenCity on the same corpus used by our MILU, denoted as UrbanFM* and OpenCity*, respectively, and evaluating them under the same zero-shot setting. MoST already uses this corpus for pretraining and is therefore included as an additional reference. This comparison allows us to examine the effect of aligning the pretraining data across methods.

MethodShort MAEShort RMSELong MAELong RMSE
UrbanFM6.55812.2708.45815.706
UrbanFM*6.26212.1797.97615.471
OpenCity9.43217.01510.94519.697
OpenCity*8.48315.80210.02018.581
MoST7.99614.05410.29917.976
MILU5.46911.1216.80213.850

The results reveal two important findings. (i) First, aligning the pretraining corpus with MILU's corpus improves both UrbanFM and OpenCity: their MAE decreases by 4.5% and 10.1% for short-term forecasting, and by 5.7% and 8.5% for long-term forecasting, respectively. This confirms that our constructed pretraining data can indeed affect performance. (ii) Second, even with the same pretraining corpus, MILU consistently outperforms UrbanFM* and OpenCity*. In particular, compared with UrbanFM*, MILU reduces MAE/RMSE by 12.7%/8.7% for short-term forecasting and 14.7%/10.5% for long-term forecasting. These results provide additional evidence that MILU's improvements are not solely attributable to its pretraining corpus, but also reflect the effectiveness of its proposed approach.

The complete comparisons and detailed discussion are in Appendix C.2 (Table 10).

W2 + Q2 — Matched inputs and architectural gains

Thanks for pointing this out. We have conducted additional matched-input experiments. Specifically, we restricted both MILU and MoST to the same four modalities: spatio-temporal observations, remote sensing, event text, and coordinates, with weather and road information disabled. With model weights fixed, we evaluated both models on the same 19 multimodal tasks. MILU reduces aggregate MAE by 27.44%/31.22% and aggregate RMSE by 16.97%/21.02% for Short/Long term forecasting, respectively, relative to MoST.

The table below reports zero-shot results for all five methods under these matched test inputs, using the same 19 tasks and test requests.

MethodShort MAEShort RMSELong MAELong RMSE
MoST10.38118.02813.53123.379
Concat7.78615.3259.79118.992
Cross-attn7.61315.1079.51818.756
FiLM7.62815.1299.40518.740
MILU7.53314.9699.30618.465

We further compared MILU against Concat, Cross-attn, and FiLM using the same backbone and modality encoders under identical multimodal inputs (Appendix C.5, Table 20). MILU consistently outperforms these alternatives. In particular, under matched remote-sensing and event inputs, it reduces aggregate MAE by 1.25%/1.05% over FiLM for Short/Long forecasting, respectively. These same-backbone comparisons provide more direct evidence of the benefits of MILU's fusion design under controlled input and architectural conditions. We have clarified the scope of the comparisons in Section 6.2 and added the matched-input results in Appendix C.2 (Table 11).

Response to Reviewer AX73 — Part 2/2

W3 + Q3 — Task-level consistency of multimodal collaboration

Joint use does not improve every task, but it consistently benefits most tasks. We analyze 17 tasks with at least two valid modalities, comparing joint use against the best-performing valid individual modality for each task. For each metric (MAE and RMSE), joint use achieves improvements on 12/17 tasks in zero-shot Short, 13/17 in zero-shot Long, 15/17 in few-shot Short, and 17/17 in few-shot Long. Note that the improved tasks may differ between MAE and RMSE.

SettingImproved tasks (MAE)Improved tasks (RMSE)
Zero-shot Short12/1712/17
Zero-shot Long13/1713/17
Few-shot Short15/1715/17
Few-shot Long17/1717/17

We also report the cases where joint use leads to regressions. For example, Melbourne flow MAE increases by 0.143% under zero-shot Long, while Toulouse occupancy MAE/RMSE increase by 0.374%/0.320% under few-shot Short. The complete task-level results, including both gains and regressions, are provided in Appendix C.6 (Table 22). We have also clarified the task-level scope of the analysis in Section 6.3.

Thank you for the thorough rebuttal and additional experiments. The shared-corpus comparisons, matched-input evaluations, and task-level analysis effectively address my main concerns regarding baseline fairness and performance attribution. Accordingly, I have raised my score.

We are glad our responses and additional experiments have addressed your concerns regarding baseline fairness and performance attribution. Thank you very much for your thoughtful and valuable feedback, which has helped us improve the work. We will incorporate all these additional experiments, analyses, and clarifications into the revised manuscript.

Again, we sincerely appreciate your time and constructive suggestions!

Reviewer_ZTgvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how road, remote-sensing, weather, and event information can support transferable urban spatio-temporal forecasting. MILU uses modality content and forecasting queries to retrieve historical states, then maps the retrieved information into a shared influence space. Aggregated influences adjust forecasting states through bounded channel scaling, supported by staged training, gain supervision, and modality-subset supervision. CityModal combines approximately 19.21 billion valid spatio-temporal data points from 87 regions across four continents with associated multimodal information. Evaluation covers short- and long-horizon forecasting on 30 tasks, including zero-shot transfer, adaptation with 10% of target training data, and unseen modality combinations.

Critical strengths

Comparing multimodal inputs enabled and disabled under the same checkpoint helps isolate the contribution of additional information. MILU achieves the lowest zero-shot MAE on 28/30 short-horizon and 27/30 long-horizon tasks, showing improvements across most evaluated tasks. CityModal aligns heterogeneous urban observations with four contextual modalities, providing a useful resource for cross-city multimodal research. Testing an unseen Road+Remote combination directly examines compositional transfer beyond combinations encountered during training. The ablations cover fusion mechanisms, training objectives, and retrieval components, providing evidence for the main design choices.

Critical weaknesses
  1. Foundation models and ablations use single training runs without uncertainty estimates, making the roughly 1% aggregate advantage over PatchSTG difficult to assess.
  2. Baselines use different pretraining corpora, released checkpoints, or reproduced implementations, leaving the contribution of architecture versus pretraining resources incompletely separated.
  3. Zero-shot multimodal RMSE gains are only 0.434% and 0.535% at the two horizons, while short-horizon occupancy RMSE worsens; evidence for reducing large errors is limited.
  4. The theory assumes independent Gaussian modalities, shared linear responses, and fixed aggregation weights, limiting its explanatory scope for nonlinear retrieval and adaptive attention.
  5. Modality scaling follows one fixed addition order, and unseen-combination testing focuses on one held-out pair, leaving broader compositional robustness uncertain.
Questions for authors
  1. How are weather reanalysis inputs filtered by actual publication time? Could valid-time alignment include information unavailable at the forecast origin?
  2. Does pretraining deduplication check shared sensors and overlapping temporal windows? How do results differ between same-city temporal transfer and entirely unseen cities?
  3. When 10% of overlapping training windows are sampled, what fraction of unique timestamps is covered? Does the stated data budget refer to windows or underlying observations?
  4. Are the differences among MILU, FiLM, and PatchSTG consistent across random seeds, particularly where reported improvements are around 1%?
  5. Do the MoST and UrbanFM reproductions recover their reported performance on their original benchmarks? Do the main gaps persist under a shared pretraining corpus?
3 comments

Response to Reviewer ZTgv — Part 1/2

Thank you for these valuable comments. We have carefully addressed your concerns through additional controlled experiments and error-distribution analyses, extensions of the theoretical analysis, and clarification of multimodal data availability and evaluation protocols. The corresponding revisions have been incorporated into the manuscript.

W1 + Q4 — Random-seed variability and the approximately 1% advantage

To address this concern, we have conducted three-seed repetitions of target-data training for MILU, FiLM, and PatchSTG. MILU and FiLM share the same pretrained initialization across repeated runs to ensure a consistent basis for comparison. The new results show that MILU maintains lower mean MAE and RMSE than PatchSTG on the 30-task aggregate and than FiLM on the 19-task multimodal aggregate.

In particular, compared with PatchSTG, MILU achieves average MAE reductions of 1.32% for short-term forecasting and 0.91% for long-term forecasting, respectively, confirming that its approximately 1% advantage persists across the three-run averages. We also observe greater seed-to-seed variability in FiLM's long-term forecasting results than in MILU's. Also, we have added the per-model means and standard deviations to Appendix C.7 (Table 27) to provide a clearer assessment of performance variability.

W2 + Q5 — Pretraining comparability and baseline implementations

Thank you for raising this concern. In this revision, we have conducted additional experiments by pretraining UrbanFM and OpenCity on the same corpus used by our MILU, while including MoST, which already uses the same corpus, as an additional reference. The results show that MILU consistently maintains its advantage over these baselines, providing further evidence that its gains are not solely attributable to pretraining resources.

In this zero-shot comparison, UrbanFM* denotes UrbanFM pretrained on MILU's corpus. In particular, compared with UrbanFM*, MILU reduces MAE/RMSE by 12.7%/8.7% for short-term forecasting and 14.7%/10.5% for long-term forecasting.

For implementation, we use the official open-source implementations of UrbanFM and MoST. For UrbanFM, we adjust the original normalization procedure to estimate statistics from visible observations, excluding masked and future targets. MoST's original pretraining dataset is not publicly released, so we train it on MILU's corpus.

We have added these results and the discussion to Appendix C.2 (Table 10). For the detailed quantitative comparisons, please refer to our response to Reviewer iSYJ, Comment W1.

W3 — RMSE gains and short-horizon occupancy degradation

Thanks for pointing this out. We would like to clarify that these results are influenced by both target-domain data characteristics and the sensitivity of RMSE to large prediction errors, and therefore should not be interpreted in isolation as a measure of the overall effectiveness of the proposed model.

Specifically, occupancy and weather distributions vary across time periods in the target data, making zero-shot transfer sensitive to local distribution shifts. To demonstrate this, we have conducted additional error-distribution analysis in Appendix C.7 (Table 24). As observed, across all four tasks exhibiting RMSE degradation, the increased squared errors among the largest-error 10% of prediction points outweigh the improvements among the remaining 90%. This explains why overall RMSE may worsen even when MAE improves, as RMSE assigns greater weight to large errors.

Importantly, after fine-tuning on target data, incorporating multimodal information reduces occupancy RMSE compared with ST-only predictions by 6.180% for short-term forecasting and 4.083% for long-term forecasting. This further indicates that multimodal information provides useful predictive signals, while its benefits in zero-shot settings can be affected by target-domain distribution shifts and the choice of evaluation metric.

Response to Reviewer ZTgv — Part 2/2

W4 — Theoretical assumptions and adaptive retrieval

Thank you for highlighting the limitations of the theoretical assumptions. To broaden its explanatory scope, we have extended the theoretical analysis in two directions.

First, we generalize the Gaussian analysis to account for correlated modalities and more general linear responses. Second, we provide an additional non-Gaussian variance comparison with concatenation under the condition that the shared subspaces preserve the underlying linear signal. The corresponding assumptions, conditions, and proofs are provided in Appendix D.3.

We also clarify the relationship between the theoretical analysis and MILU's actual architecture. The theoretical risk analysis focuses on fixed linear representations and is intended to characterize the benefits of multimodal aggregation under tractable assumptions, rather than directly model the entire nonlinear retrieval process. In MILU, nonlinear retrieval is followed by a shared linear mapping, and the retrieved representations from available modalities are combined through normalized summation (Appendix D.1). Thus, the theoretical analysis provides insights into the aggregation mechanism, while the nonlinear retrieval component is handled by the model architecture.

W5 — Modality addition order and unseen combinations

In this rebuttal, we conducted additional experiments to examine the robustness of MILU under more diverse modality configurations in Appendices C.4 and C.6 (Tables 16 and 23).

First, we evaluated all 16 modality subsets using fixed model weights, covering all 32 single-modality addition transitions. The aggregate MAE and RMSE do not increase at either forecasting horizon across these transitions, indicating that the performance benefits are not sensitive to a particular modality addition order. Second, to broaden the evaluation of compositional generalization, we added a Road+Weather holdout setting alongside the original Road+Remote holdout. In both settings, the unseen modality pair outperforms its constituent single modalities in both MAE and RMSE at both forecasting horizons. In particular, Road+Weather achieves additional MAE reductions of 0.515% for short-term forecasting and 0.202% for long-term forecasting over the better-performing constituent modality.

Q1 — Weather publication time

We agree that data publication schedules and update latencies vary across modalities and should be explicitly considered. Accordingly, we revised our original claim to clarify that MILU follows modality-specific temporal availability constraints, rather than assuming all multimodal information is immediately accessible before prediction, to avoid potential misunderstandings.

We have addressed the issue through additional experiments using operational GFS forecasts.

For each forecast origin, we selected the latest GFS run that was recorded or reconstructed to have become available before that origin. All model weights and other input modalities were kept unchanged, thereby isolating the impact of weather-data availability.

Across 15 tasks in seven datasets, replacing the original weather inputs with operationally available GFS forecasts results in only marginal performance changes: MAE/RMSE increase by 0.0281%/0.0160% for short-term forecasting and 0.0397%/0.0268% for long-term forecasting.

Please refer to our response to Reviewer iSYJ, Comment W2, for the detailed methodology, results, and revisions.

Q2 — Pretraining overlap and transfer settings

Our deduplication checks both shared sensors and overlapping input–target windows. We identified overlapping PEMS04/PEMS07 flow observations across 19 test datasets and excluded these observations and all affected windows, including those in neighboring spatial groups (Appendix C.1).

Compared with same-corpus UrbanFM, MILU achieves larger relative MAE reductions in same-city temporal transfer (California 2024: 6.98%/6.42% for Short/Long) than in unseen-city transfer (San Diego: 3.36%/2.59%). RMSE also improves in both settings, with smaller gains on the unseen city (Appendix C.7, Table 26).

Q3 — The 10% data budget and overlapping windows

We have clarified the 10% target-data budget and corrected the description of the sampling procedure.

We first select temporal blocks covering at most 10% of the target training time points, and then construct complete input–target windows exclusively within the selected blocks. Therefore, overlapping windows do not introduce additional time points outside the selected blocks, and the 10% constraint is maintained at the time-point level. We have added the corresponding details to Appendix C.1.

Please refer to our response to Reviewer iSYJ, Comment M2, for the detailed explanation and corresponding manuscript revisions.

As the rebuttal period is approaching its deadline, thank you again for your thoughtful review and constructive feedback. We would like to briefly summarize the additional experiments, evidence, and clarifications provided in our rebuttal:

  • W1 / Q4: We have added three-run results with means and standard deviations to assess the stability of the differences among MILU, FiLM, and PatchSTG.

  • W2 / Q5: We have pretrained UrbanFM and OpenCity on the same corpus as MILU, enabling a more controlled comparison of pretraining data, and have clarified the baseline implementations. This additional evaluation has also been acknowledged by Reviewer AX73.

  • W3: We have added an error-distribution analysis and target-adaptation results to clarify the observed zero-shot RMSE degradation.

  • W4: We have extended the analysis to correlated modalities and a non-Gaussian setting, and have clarified its assumptions and relationship to MILU's nonlinear retrieval.

  • W5: We have evaluated all modality subsets and addition transitions, and have added a second held-out modality pair.

  • Q1–Q3: We have evaluated operational GFS forecasts and have clarified pretraining deduplication, same-city versus unseen-city transfer, and the time-point-level 10% data budget.

We would be grateful for any further questions or feedback and would be happy to provide additional clarification.

Reviewer_iSYJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed MILU, a multimodal model that forecasts spatio-temporal data by learning each modality's predictive influence. The authors also constructed a multimodal dataset for training the model. Then the authors trained the model using the constructed dataset. The evaluation includes zero-shot, few-shot forecasting and other experiments like abalation study, scaling analysis. It shows that the proposed method remains competitive with baselines.

Critical strengths
  1. Forecasting spatio-temporal data by learning each modality's predictive influence is an interesting and important research topic. Modality-conditioned historical retrieval and bounded feature modulation provide a coherent design for incorporating external information into spatio-temporal forecasting.

  2. The evaluation covers zero-shot transfer, few-shot, target-data adaptation, and unseen modality combinations. The on/off comparisons and component ablations are useful for evaluating the contribution of multimodal inputs and design choices.

Critical weaknesses
  1. The largest concern is that different methods use different data for training, which makes it difficult to evaluate whether the gain is from the training dataset or the proposed method. For example, MILU ‘pretraining uses CityModal supplemented with public datasets‘ (line 356) while MoST ‘pretrained MoST architecture with multimodal inputs‘ and UrbanFM ‘Pretrained on the public WorldST package’ (line 1120). A controlled comparison with reproduced baselines using the same training dataset and comparable training budgets would strengthen the evaluation. I suggest the authors use the same dataset to train baselines when doing the implementation.
  2. The claim of ‘multimodal information available before the forecast origin’ is not well established. It limits the value for real-world downstream applications. For example, the ERA5 data is updated daily, with a latency of about 5 days behind real time. Different datasets have different latency. They author may consider the difference for input.
Questions for authors

See the questions in Weaknesses.

More:

  1. The quality of Figure 1 needs to be improved. The figure should be self-contained. From the figure, I have no idea about MoST and FiLM in (iv), and I cannot find any explanation in the caption when reading this figure.
  2. For the few-shot training setting, the authors claimed to use 10% target data. However, if the setting uses 24-to-24 training windows, then more than 90% of the data points will be used at least once (1-0.9^24). The author needs to provide more details about the setting and modify the claim if needed.
2 comments

Response to Reviewer iSYJ — Part 1/2

Thank you for these valuable comments. We have carefully addressed all four concerns through additional controlled experiments, clarification/validation of multimodal data availability, and improvements to the presentation and description. The corresponding revisions have been incorporated into the manuscript.

W1 — Pretraining data and budget comparability

Following the suggestion, we have conducted additional controlled experiments by pretraining UrbanFM and OpenCity on the same corpus used by our MILU, denoted as UrbanFM* and OpenCity*, respectively, and evaluating them under the same zero-shot setting. MoST already uses this corpus for pretraining and is therefore included as an additional reference. This comparison allows us to examine the effect of aligning the pretraining data across methods.

MethodShort MAEShort RMSELong MAELong RMSE
UrbanFM6.55812.2708.45815.706
UrbanFM*6.26212.1797.97615.471
OpenCity9.43217.01510.94519.697
OpenCity*8.48315.80210.02018.581
MoST7.99614.05410.29917.976
MILU5.46911.1216.80213.850

The results reveal two important findings. (i) First, aligning the pretraining corpus with MILU's corpus improves both UrbanFM and OpenCity: their MAE decreases by 4.5% and 10.1% for short-term forecasting, and by 5.7% and 8.5% for long-term forecasting, respectively. This confirms that our constructed pretraining data can indeed affect performance. (ii) Second, even with the same pretraining corpus, MILU consistently outperforms UrbanFM* and OpenCity*. In particular, compared with UrbanFM*, MILU reduces MAE/RMSE by 12.7%/8.7% for short-term forecasting and 14.7%/10.5% for long-term forecasting. These results provide additional evidence that MILU's improvements are not solely attributable to its pretraining corpus, but also reflect the effectiveness of its proposed approach.

The complete comparisons and detailed discussion are in Appendix C.2 (Table 10).

W2 — Multimodal availability before the forecast origin

Thank you for pointing this out. We agree that data publication schedules and update latencies vary across modalities and should be explicitly considered. Accordingly, we revised our original claim to clarify that MILU follows modality-specific temporal availability constraints, rather than assuming all multimodal information is immediately accessible before prediction, to avoid potential misunderstandings.

Specifically, road networks and satellite imagery are treated as static contextual information, whereas event data are filtered based on both their actual availability time and validity period. More importantly, to account for the publication latency of operational weather products such as ERA5 (which typically becomes available several days after real time), we have conducted additional experiments using operational Global Forecast System (GFS) forecasts. For each forecast origin, we selected the latest GFS run that was recorded or reconstructed to have become available before that origin. All model weights and other input modalities were kept unchanged, thereby isolating the impact of weather-data availability.

Across 15 tasks in seven datasets, replacing the original weather inputs with operationally available GFS forecasts results in only marginal performance changes: MAE/RMSE increase by 0.0281%/0.0160% for short-term forecasting and 0.0397%/0.0268% for long-term forecasting. We also conducted a consistency analysis showing good agreement between GFS and the original weather inputs across several weather variables. These additional results demonstrate that MILU can effectively leverage operationally available weather forecasts, with negligible accuracy degradation on the evaluated tasks, rather than relying solely on retrospectively available weather observations.

We have clarified the modality-specific temporal availability rules in Appendix B.2 and added the operational GFS evaluation and consistency analysis to Appendix C.7 (Table 25).

Response to Reviewer iSYJ — Part 2/2

M1 — Figure 1 clarity

Thanks for the comment. Specifically, we have revised its caption to briefly explain FiLM and MoST with the corresponding references, identify the methods illustrated in panels (i)–(iii), and clarify the performance reference used in panel (iv). These minor revisions improve the readability of Fig. 1 (Page 2).

M2 — Few-shot sampling and the 10% data budget

Thank you for pointing out the ambiguity. We have corrected this description and clarified the sampling procedure: we first select temporal blocks covering at most 10% of the target training time points, and then construct complete input–target windows exclusively within the selected blocks. Therefore, overlapping windows do not introduce additional time points outside the selected blocks, and the 10% constraint is maintained at the time-point level. We have added the corresponding details to Appendix C.1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback. We summarize the main updates below.

  • Baseline comparability. To test whether MILU’s gains persist under comparable conditions, we pretrained UrbanFM and OpenCity on MILU’s corpus and matched the test inputs of MILU and MoST. MILU achieved lower aggregate MAE and RMSE in both comparisons.

  • Input availability and data budget. To assess forecast-time availability, we evaluated operational GFS forecasts and documented public availability dates for road networks and satellite imagery. Aggregate errors changed only slightly with GFS. We also clarified pretraining overlap checks and the 10% cap on target training time points.

  • Multimodal collaboration. We added task-level comparisons, evaluated all 16 modality subsets, and tested a second held-out pair. Joint use outperformed the best individual modality on most tasks with at least two valid modalities, and the held-out Road+Weather pair outperformed either modality alone.

We hope this summary is helpful in your assessment of our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.