acceptodds
Under review as a conference paper at ICLR 2027

From AI Forecast to Text Report: Pangu-Driven Weather Report Generation via Vision-Language Fine-tuning

Abstract

Recent work, WeatherSyn showed MLLMs can generate them from ERA5 reanalysis images, but ERA5 has multi-day latency. More fundamentally, it is produced via 4D-Var assimilation over a 12-hour window thus using it as model input for real-time forecasting can leak future information and inflate apparent skill. Another work, WeatherQA, by contrast, targets severe-weather reasoning and text-only discussion generation, not forecast heatmaps, thereby omitting critical spatial information, limiting operational utility. In this paper, we ask whether AI-based weather forecast fields, specifically from Pangu-Weather, can replace ERA5 as the visual input for driving text report generation. We construct an end-to-end pipeline that renders city-level meteorological heatmaps for 31 U.S. cities, pairs them with professional NWS forecast discussions from the WSInstruct corpus, and LoRA fine-tunes Qwen2-VL-7B on 2,016 paired samples from 2020–2021. Our central contribution is a controlled input-source ablation: we train two models under identical hyperparameters, differing only in whether the visual input is rendered from Pangu forecast fields or ERA5 reanalysis fields, and evaluate both on both test sets. On 1,043 test samples from 2022, the input-source cost of replacing ERA5 with Pangu is modest—at most ROUGE-L and BLEU-1—and training on Pangu forecast fields does not hurt downstream report generation. These results indicate that end-to-end, real-time AI weather forecasting pipelines, from initial conditions to human-readable reports, are feasible without relying on delayed reanalysis products. This is the first systematic study of whether AI forecast fields can replace reanalysis in driving text weather report generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Pangu, AI, Forecast, Report, Weather, Vision-Language, Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SCQ9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores how to automatically generate weather reports using AI. The authors first use Pangu-Weather to predict weather conditions 24 hours ahead and convert the predictions into weather images showing temperature, pressure, wind speed, and other information. These images are then used to train a vision-language model, Qwen2-VL-7B, to generate natural-language weather reports. The authors conduct experiments using data from 31 U.S. cities and compare the results of using Pangu forecasts with those of using ERA5 data. The experimental results show that both approaches achieve similar BLEU and ROUGE scores. Based on these findings, the authors conclude that Pangu forecasts can be used as an alternative to ERA5 data for automatic weather report generation.

Critical strengths
  1. The paper explores how to generate natural-language weather reports from AI-based weather forecasts. This is a practically valuable problem, as it could help reduce the manual effort required to produce weather reports.

  2. The paper presents a straightforward pipeline that uses Pangu-Weather to generate weather forecasts, converts the forecast data into heatmaps, and then uses a vision-language model to generate weather reports. The comparison between Pangu and ERA5 inputs is also a reasonable experimental design.

Critical weaknesses
  1. Although the paper discusses several related studies, it does not sufficiently explain how its approach differs from existing methods. In particular, previous studies have already explored using vision-language models to generate weather reports. The authors should provide a more comprehensive discussion of related work and clearly explain the unique contributions of this paper.

  2. The paper mainly uses BLEU and ROUGE scores to evaluate the generated reports. However, these metrics primarily measure textual similarity rather than the factual accuracy of weather forecasts. For example, "Heavy rainfall is expected tomorrow" and "Heavy rainfall is not expected tomorrow" share many words but convey completely different meanings. Therefore, the authors should include additional evaluation metrics that measure meteorological accuracy, rather than relying solely on textual similarity.

  3. Figure 1 only presents a basic overview of the pipeline. While it shows the general workflow, it does not clearly explain how Pangu and ERA5 inputs are compared, even though this comparison is the central contribution of the paper. The authors should improve the figure to better illustrate the data processing steps, model training process, and experimental comparison.

  4. The proposed method feeds weather heatmaps into a vision-language model to generate reports. However, it is unclear whether the model genuinely extracts useful meteorological information from these images or simply learns common patterns in weather report writing. The authors should conduct additional ablation studies, such as generating reports without images or with randomly selected weather images. If the model achieves similar performance without meaningful visual inputs, it would be difficult to conclude that the weather images play an important role in report generation.

  5. The proposed method relies on a relatively complex vision-language model. However, if the goal is to generate text based on weather variables such as temperature, wind speed, and atmospheric pressure, simpler rule-based or template-based approaches might also be effective. For example, a template could generate statements such as "The maximum temperature tomorrow will be XX°C" directly from numerical weather data. The authors should compare their method against simple rule-based or template-based baselines to demonstrate whether the proposed approach produces more accurate, informative, or useful weather reports.

Questions for authors
  1. How can we know whether the model actually understands the weather images? Have the authors tried generating reports without images or with random images to see how the results change?

  2. How can the authors make sure that the generated weather reports are actually correct? Could the authors add an evaluation of weather information accuracy?

  3. Why is a complex vision-language model needed for this task? Could a simple rule-based or template-based method generate similar weather reports? Have the authors compared their method with such simpler approaches?

  4. The paper claims that the proposed method can generate weather reports in real time. However, Pangu-Weather still uses ERA5 data as its initial input. How can the system work in real time if ERA5 data is delayed?

Reviewer_bvqMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether Pangu-Weather forecast fields can replace ERA5 reanalysis images as inputs to weather-report generation. It produces 24-hour forecasts, renders four meteorological variables for 31 U.S. locations, and LoRA-fine-tunes Qwen2-VL-7B. Two models share 2,016 reported training examples from 2020–2021 and are each evaluated with both input sources on 1,043 reported 2022 examples. The resulting 2×2 comparison yields similar BLEU-1 and ROUGE-L scores.

Critical strengths

Appendices A.3–A.8 provide the actual instruction, field transformations, LoRA settings, deterministic decoding, and metric implementations. These details substantially improve inspectability.

Critical weaknesses
  1. The central information-preservation conclusion is not established by the evaluation presented. Section 5.2 interprets small BLEU-1/ROUGE-L differences as evidence of sufficient weather information. Section 4.8 states that city-level variation and domain-specific weather phenomenon matching are evaluated, but no corresponding results are provided in Section 5 or the appendices; aspect-level weighted F1 is listed as future work in Section 6. No image-free or image-shuffled control or uncertainty analysis is presented either. Similar lexical scores could arise from location/language priors or metrics insensitive to timing, negation, and severity. The reported evidence does not distinguish these explanations, which is consequential to the claimed usefulness of the forecast images.
  2. Temporal provenance and the real-time interpretation remain unresolved. Section 4.1 describes initialization data from 2021 only, whereas Table 2 reports 2020–2021 training and 2022 testing. Initialization, forecast-valid time, comparison-field time, and discussion horizon are not unambiguously reconciled. Appendix A.5 still uses ERA5 reanalysis initialization, so shared retrospective inputs do not demonstrate operation ‘without relying on delayed reanalysis products.’
Questions for authors
  1. Please provide a representative pairing audit with initialization UTC, forecast-valid UTC, ERA5 comparison UTC, NWS issuance UTC, target horizon, and sample counts for each claimed year. Appendix A.3 already gives a city-only prompt template, and Appendix A.7 states that inference uses the same textual prompt as training. Please confirm whether this was the template actually used, provide a representative target report, and reconcile it with the single-day target in Section 3.1.2 and the multi-day, explicitly aspect-conditioned description in Sections 4.5–4.6.
  2. Please provide the definitions and results of the weather-phenomenon matching evaluation mentioned in Section 4.8. Can these results, together with an image-free or image-shuffled control and a targeted weather-fact audit, distinguish preserved information from visual-input insensitivity on the existing test set? Please report paired uncertainty, accounting for shared dates/locations and training variability where training-source benefits are claimed.
  3. Which initialization product was available at the asserted issuance time? If only retrospective ERA5 was used, should the conclusion be restricted to offline feasibility?
  4. Please clarify the input-switch direction and unrounded differences in Table 1. From the displayed values, ERA5-to-Pangu inputs change the ERA5-trained model by −0.0006 BLEU-1/−0.0004 ROUGE-L, but improve the Pangu-trained model by +0.0024/+0.0033; the larger stated ‘cost’ describes the reverse switch.
Reviewer_kAghOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses Pangu-Weather to produce 24-hour forecasts, convert four meteorological variables into heatmaps, and fine-tune Qwen2-VL-7B with LoRA to generate natural-language weather reports.

Critical strengths
  • Using actual forecast fields instead of delayed reanalysis data is a reasonable direction for real-world weather report generation.

  • The 2×2 comparison between Pangu and ERA5 is straightforward and helps isolate the effect of different input sources.

  • The paper provides fairly detailed descriptions of the data processing pipeline, training configurations, and inference settings.

Critical weaknesses
  • The paper mostly combines existing components (Pangu-Weather, Qwen2-VL, and LoRA) without introducing a new method. WeatherSyn has also explored using forecast fields for report generation. It's hard to see what new technical insight this paper brings beyond swapping ERA5 for Pangu.
  • Pangu still takes ERA5 as its initial input in this work. So the data latency issue, which is one of the main motivations of the paper, is not really solved.
  • The evaluation is too weak. BLEU-1 and ROUGE-L are not enough to evaluate weather reports. Two reports can have similar wording but describe completely different weather conditions. There is no evaluation of factual weather accuracy, and the reported improvements are very small without any statistical analysis.
  • Could the authors clarify the dataset construction and temporal alignment? In particular, how are the 2020 and 2022 samples obtained given the description in Section 4.1?
Questions for authors

Please see the above comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.