acceptodds
Withdrawn from ICLR 2027

Learning from Multi-Perspectives: A Multimodal Precipitation Nowcasting Framework

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Precipitation nowcasting is a crucial spatio-temporal forecasting task with broad applications in weather warning, transportation, and agriculture. Existing deep learning based methods typically model precipitation evolution either from a radar-native perspective that captures signal-level motion and intensity dynamics, or from a visual perspective that extracts high-level morphological structures of precipitation system. Although these two perspectives are complementary, few works have integrated them into a unified framework. Moreover, recent works have introduced textual meteorological priors but only couple them with visual representations, leaving their potential synergy with radar-native dynamics largely unexplored. In this paper, we propose PreMM, a multimodal precipitation nowcasting framework that jointly models radar-native dynamics, visual representations, and textual semantic priors. PreMM consists of a Radar Dynamics Learner (RDL) that explicitly decouples radar evolution into motion and intensity components, and enhances them with frequency-domain and textual priors; a Visual Perception Learner (VPL) that reconstructs radar observations into visual representations with frequency and periodicity cues, and leverages a pre-trained vision-language encoder to capture precipitation structures; a Text Prior Generator (TPG) that produces meteorological semantic priors; and a Multimodal Fusion Network (MFN) that integrates these complementary representations for future radar prediction. Extensive experiments on multiple precipitation nowcasting benchmarks demonstrate that PreMM consistently outperforms state-of-the-art methods.

Primary area:
learning on time series and dynamical systems
Keywords:
Precipitation Nowcasting, Multimodal, Multi-Perspectives

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kjRrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper utilizes large-scale model prediction of text for precipitation prediction. It integrates three perspectives—radar native dynamics, visual representation, and text semantic priors—into a unified framework and designs four modules: RDL, VPL, TPG, and MFN. It achieves state-of-the-art results on four datasets: SEVR, CIKM, and Shanghai.

Critical strengths
  1. This work integrates radar native dynamics, visual representation, and textual semantic priors into a single framework for nowcasting, overcoming the shortcomings of current "single-viewpoint" prediction methods.

  2. A significant advantage lies in using frame difference + phase prior and statistical descriptor + amplitude prior for separate modeling, followed by cross-attention to fuse intensity and motion.

  3. The metrics are also generally leading.

Critical weaknesses

Serious issues:

  1. The comparison method is outdated and needs to be updated. It's recommended to add new methods for comparison based on data from 2025-2026.

  2. The ablation experiments are insufficient. They don't demonstrate the benefits of "motion/intensity decoupling," relying instead on "dual cross-attention" or even simple methods like "frame difference." More precise ablation is recommended.

Other issues: 3. Zero-parameter positional coding. I didn't fully understand this part. It seems similar to positional coding to me. The author should revisit this section.

Questions for authors

See Weaknesses

Reviewer_tqneOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposed PreMM, a multimodal precipitation nowcasting framework that jointly models radar-native dynamics, visual representations, and textual semantic priors. It contains three key designs: Radar Dynamics Learner, Vision Perception Learner, and Text Prior Generator. The authors then evaluated the method against 10 baselines on 4 datasets. The results show that PreMM outperforms the baselines.

Critical strengths
  1. The paper addresses precipitation nowcasting, an important problem. Combining radar dynamics, visual representations, and textual priors provides a reasonable motivation for the proposed framework.

  2. The evaluation covers four precipitation nowcasting datasets and includes both event-based and continuous-field metrics. The results show that the proposed method performs better than other methods.

Critical weaknesses
  1. The experiments do not sufficiently support the claimed mechanisms. For example, the paper claims that RDL 'explicitly decouples radar evolution into motion and intensity dynamics' (line 74), but removing RDL entirely only demonstrates the utility of the overall branch. It does not establish that the learned representations disentangle motion and intensity. The authors claim 'semantic priors from the VLM can provide effective guidance' (line 422) by removing TPG entirely. However, the ablation study does not isolate the contribution of meteorological semantics. Comparisons with fixed random embeddings and general-domain text, while keeping the fusion architecture unchanged, would help support this claim.
  2. There are many inconsistent implementation details. For example, 'use CLIP as the default vision-language model' (line 653). But the main result uses Qwen2.5-VL. And the learning rate: 'initial learning rate of 1 × 10−4' (line 653) and 10-3 (Table 4). The batch sizes also differ between these sections.
Questions for authors

See the questions in Weaknesses.

More:

  1. There are too many formulas while it is not a paper with theory contribution. Some formulas are not necessary. For example: (4) how to calculate mean and std. (8)(9) attention calculation. (15) image normalization. The authors are suggested to review these formulas and keep only the key definitions in the manuscript.

  2. Why are there two names for the proposed method: PreMM and PreVLM(fig. 2, fig. 3)?

  3. There is no required AI usa stagement section.

Reviewer_5gwLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PreMM, a multimodal precipitation nowcasting framework that combines radar-native dynamics, visual representations, and textual priors. Its Radar Dynamics Learner (RDL) models motion and intensity through separate branches, incorporating frequency-domain and textual priors and bidirectional cross-attention. Its Visual Perception Learner (VPL) transforms radar observations into visual representations and uses a pretrained vision–language encoder to extract high-level precipitation structures. A Text Prior Generator (TPG) and Multimodal Fusion Network (MFN) provide semantic guidance and integrate the representations. Experiments on four benchmarks report competitive performance across multiple metrics.

Critical strengths
  1. Promising multi-perspective modeling. Combining domain-specific radar dynamics with pretrained visual representations offers a valuable direction for integrating different levels of information in precipitation nowcasting.
  2. Reasonable motion–intensity interaction design. Modeling motion and intensity through separate branches with bidirectional cross-attention is a reasonable approach to capturing their interactions.
  3. Competitive empirical results. PreMM performs well across multiple metrics on SEVIR, MeteoNet, Shanghai, and CIKM, particularly at some high-intensity thresholds.
Critical weaknesses
  1. The motivation for multi-perspective fusion is insufficiently established.

The paper moves from the underexplored interaction between textual priors and radar-native dynamics to the need for a unified radar–vision–text framework. However, this identifies an unexplored combination rather than a specific forecasting problem caused by its absence.

The claimed complementarity between fine-grained radar dynamics and high-level visual structures is also insufficiently explained. It remains unclear where existing methods fail and why additional visual representations and textual priors would address those failures.

In particular, the paper does not clarify what meteorological knowledge existing methods lack or why encoding it as text should reduce noise sensitivity or improve motion and intensity modeling.

  1. The experiments do not sufficiently validate the core designs and their proposed mechanisms.

(1) The semantic contribution of textual priors remains unclear. The priors encode predefined prompts shared across radar sequences rather than event-specific dynamics. Encoding an instruction to describe radar motion does not itself provide a motion description of the current sequence. Existing ablations do not distinguish gains from meteorological semantics from those of additional fixed conditioning representations.

(2) Important controlled comparisons are missing. A baseline combining the same radar backbone and frozen visual encoder through simple fusion would help establish whether the proposed designs offer benefits beyond standard pretrained-feature integration. LangPrecip is also a key reference in the motivation but is absent from the main comparisons, limiting the assessment of PreMM’s relative advantages.

(3) Mechanism-level interpretability is limited. Module ablations and prediction visualizations do not establish whether the components fulfill their intended roles: whether RDL learns distinct motion and intensity information, whether VPL captures structural information beyond radar dynamics, and whether textual priors provide meaningful semantic guidance. The results primarily demonstrate module-level contributions rather than explain their mechanisms.

(4) Computational efficiency is insufficiently evaluated. The framework introduces pretrained vision–language components and multiple processing branches, but lacks a complete efficiency comparison. Model-size experiments relate parameter count to accuracy without reporting training cost, inference latency, or GPU memory usage, leaving the performance–cost trade-off unclear.

Questions for authors
  1. Which specific limitations of existing radar-native and visual methods motivate jointly incorporating visual representations and textual priors rather than using simpler feature fusion?
  2. Could the authors compare meteorological text embeddings with fixed random or learnable embeddings of matched dimensions and comparable norms to isolate the contribution of semantics?
  3. Could the authors provide mechanism-level analyses showing what the motion, intensity, and visual branches learn, how textual guidance operates, and how these components improve predictions?
  4. Could the authors include a simple radar–vision fusion baseline and consider a comparison with LangPrecip to better establish the benefits of the proposed framework?
  5. Could the authors report training cost, inference latency, and GPU memory usage under comparable settings to assess the performance–cost trade-off against existing methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.