- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MILU, a multimodal foundation model for urban spatio-temporal forecasting that addresses interference among heterogeneous modalities. The method introduces modality-conditioned historical retrieval, shared influence learning, and bounded feature modulation, together with a multimodal pretraining strategy designed to improve cross-city transferability. The authors also construct CityModal, a large-scale dataset covering 87 geographic regions and approximately 19.21 billion valid spatio-temporal data points. Extensive experiments demonstrate strong zero-shot forecasting performance, effective few-shot adaptation, and generalization to unseen modality combinations.
Critical strengths
-
Well-motivated methodology and strong empirical performance.The proposed shared influence learning framework provides a meaningful alternative to conventional multimodal fusion by explicitly addressing interference among heterogeneous modalities. MILU achieves the best MAE on 28/30 short-term and 27/30 long-term zero-shot forecasting tasks. The extensive experiments, including few-shot adaptation and ablation studies, provide convincing evidence of its effectiveness and transferability.
-
Valuable large-scale dataset contribution. CityModal integrates diverse multimodal information across 87 geographic regions, containing approximately 19.21 billion valid spatio-temporal data points. Its substantial scale, modality diversity, and geographic coverage provide a valuable resource for future research on multimodal urban spatio-temporal foundation models.
Critical weaknesses
- Differences in pretraining data across baselines. As described in Appendix C.2, different foundation-model baselines use different pretraining corpora or publicly released checkpoints. Therefore, the zero-shot performance improvements in Table 2 may reflect both methodological advantages and differences in pretraining data. A clearer discussion of this distinction would strengthen the experimental conclusions.
- Comparability of multimodal input conditions. Section 6.2 compares models using their individually supported input modalities. Since some baselines do not utilize the same multimodal information as MILU, part of the performance difference may arise from differences in input availability. Although the authors provide controlled ablation experiments, the interpretation of these comparisons could be further clarified.
- Limited fine-grained analysis of multimodal collaboration. Figures 1 and 6 demonstrate encouraging aggregate improvements when multiple modalities are combined. However, these aggregated results do not fully reveal whether the proposed method consistently avoids negative interactions across individual forecasting tasks. Additional task-level analysis of joint-use gains relative to the best individual modality would further support the central motivation.
If the authors can satisfactorily address these concerns, I would be willing to increase my overall rating.
Questions for authors
-
Pretraining Data Comparability: We noticed in Appendix C.2 that several baseline models use publicly released pretrained weights or are pretrained on different datasets (e.g., UrbanFM on WorldST), rather than being pretrained on the same corpus as MILU. Consequently, there may be substantial differences in pretraining data scale, diversity, and geographic coverage between MILU and these baselines. These differences could potentially contribute to the performance gaps observed in the zero-shot experiments in Table 2, making it difficult to attribute the improvements solely to the proposed methodology. If possible, additional evidence under more comparable pretraining conditions would further strengthen the conclusions.
-
Fairness of Multimodal Comparisons: Since different baselines support different input modalities, could the authors clarify how the fairness of multimodal comparisons is ensured? Are there additional controlled comparisons under matched input conditions, or could the authors further distinguish gains from architectural improvements and those from access to additional modalities?
-
Consistency of Multimodal Collaboration: The aggregate results suggest that combining multiple modalities improves forecasting performance. However, does this improvement consistently hold across individual forecasting tasks? Could the authors provide a more detailed analysis of joint-use gains relative to the best-performing individual modality, particularly highlighting cases where multimodal combination fails to provide additional benefits?
4 comments
Response to Reviewer AX73 — Part 2/2
W3 + Q3 — Task-level consistency of multimodal collaboration
Joint use does not improve every task, but it consistently benefits most tasks. We analyze 17 tasks with at least two valid modalities, comparing joint use against the best-performing valid individual modality for each task. For each metric (MAE and RMSE), joint use achieves improvements on 12/17 tasks in zero-shot Short, 13/17 in zero-shot Long, 15/17 in few-shot Short, and 17/17 in few-shot Long. Note that the improved tasks may differ between MAE and RMSE.
| Setting | Improved tasks (MAE) | Improved tasks (RMSE) |
|---|---|---|
| Zero-shot Short | 12/17 | 12/17 |
| Zero-shot Long | 13/17 | 13/17 |
| Few-shot Short | 15/17 | 15/17 |
| Few-shot Long | 17/17 | 17/17 |
We also report the cases where joint use leads to regressions. For example, Melbourne flow MAE increases by 0.143% under zero-shot Long, while Toulouse occupancy MAE/RMSE increase by 0.374%/0.320% under few-shot Short. The complete task-level results, including both gains and regressions, are provided in Appendix C.6 (Table 22). We have also clarified the task-level scope of the analysis in Section 6.3.
Thank you for the thorough rebuttal and additional experiments. The shared-corpus comparisons, matched-input evaluations, and task-level analysis effectively address my main concerns regarding baseline fairness and performance attribution. Accordingly, I have raised my score.
We are glad our responses and additional experiments have addressed your concerns regarding baseline fairness and performance attribution. Thank you very much for your thoughtful and valuable feedback, which has helped us improve the work. We will incorporate all these additional experiments, analyses, and clarifications into the revised manuscript.
Again, we sincerely appreciate your time and constructive suggestions!
Response to Reviewer AX73 — Part 1/2
Thanks for these valuable comments. We have conducted additional controlled experiments and task-level analyses to better clarify the sources of MILU's performance gains, ensure fairer comparisons across different input conditions, and examine the consistency of multimodal collaboration. We have also clarified the scope and interpretation of the corresponding experimental results in the revised manuscript.
W1 + Q1 — Pretraining data comparability
To address this concern, we have conducted additional controlled experiments by pretraining UrbanFM and OpenCity on the same corpus used by our MILU, denoted as UrbanFM* and OpenCity*, respectively, and evaluating them under the same zero-shot setting. MoST already uses this corpus for pretraining and is therefore included as an additional reference. This comparison allows us to examine the effect of aligning the pretraining data across methods.
The results reveal two important findings. (i) First, aligning the pretraining corpus with MILU's corpus improves both UrbanFM and OpenCity: their MAE decreases by 4.5% and 10.1% for short-term forecasting, and by 5.7% and 8.5% for long-term forecasting, respectively. This confirms that our constructed pretraining data can indeed affect performance. (ii) Second, even with the same pretraining corpus, MILU consistently outperforms UrbanFM* and OpenCity*. In particular, compared with UrbanFM*, MILU reduces MAE/RMSE by 12.7%/8.7% for short-term forecasting and 14.7%/10.5% for long-term forecasting. These results provide additional evidence that MILU's improvements are not solely attributable to its pretraining corpus, but also reflect the effectiveness of its proposed approach.
The complete comparisons and detailed discussion are in Appendix C.2 (Table 10).
W2 + Q2 — Matched inputs and architectural gains
Thanks for pointing this out. We have conducted additional matched-input experiments. Specifically, we restricted both MILU and MoST to the same four modalities: spatio-temporal observations, remote sensing, event text, and coordinates, with weather and road information disabled. With model weights fixed, we evaluated both models on the same 19 multimodal tasks. MILU reduces aggregate MAE by 27.44%/31.22% and aggregate RMSE by 16.97%/21.02% for Short/Long term forecasting, respectively, relative to MoST.
The table below reports zero-shot results for all five methods under these matched test inputs, using the same 19 tasks and test requests.
We further compared MILU against Concat, Cross-attn, and FiLM using the same backbone and modality encoders under identical multimodal inputs (Appendix C.5, Table 20). MILU consistently outperforms these alternatives. In particular, under matched remote-sensing and event inputs, it reduces aggregate MAE by 1.25%/1.05% over FiLM for Short/Long forecasting, respectively. These same-backbone comparisons provide more direct evidence of the benefits of MILU's fusion design under controlled input and architectural conditions. We have clarified the scope of the comparisons in Section 6.2 and added the matched-input results in Appendix C.2 (Table 11).