acceptodds
Under review as a conference paper at ICLR 2027

MeteoSage: An Operational Weather Agent Connecting Heterogeneous Observations to Forecasts and Service

Abstract

Operational weather forecasting requires synthesizing heterogeneous observations and specialized forecast products to assess evolving weather hazards and deliver timely public services. Although AI has advanced individual forecasting tasks, integrating these capabilities into a coherent, traceable workflow remains challenging because of heterogeneous data semantics, conflicting evidence, and context-dependent forecasting strategies. We introduce , an agentic framework connecting observations, analysis, forecasting, and weather services through three complementary components: (i)  provides unified access to heterogeneous data and controlled tool execution; (ii)  links evolving assessments and unresolved discrepancies to the underlying observations to guide conflict assessment and resolution; and (iii)  distills eligible task records and feedback into reusable, context-specific guidance. We further introduce WxFlowBench, a benchmark built from multi-source observations, operational forecast products, and real-world meteorological reports. Its five tracks cover evidence-grounded analysis (T0), nowcasting over 0–3 h (T1), forecasting from 3 h to 15 d (T2), meteorological report generation (T3), and public weather services and emergency support (T4). Comprehensive evaluation on WxFlowBench shows that performance superior across all tasks. Our analysis identifies distinct failure modes involving forecast-model selection, sensor-modality mismatches, and forecast-pipeline composition. Dataset and code: https://github.com/useliz/MeteoSage.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
weather agent, earth observations, operational weather forecasting, decision support

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3tFoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how AI agents can combine heterogeneous meteorological evidence and forecasting tools to produce traceable analyses, forecasts, and service outputs. It addresses three challenges: differences in data semantics, conflicting evidence, and forecasting strategies that depend on the task context. MeteoSage combines three components: a Meteorological Agentic Environment that preserves data definitions and provides scientific tools; an Event-Centered Evidence Workspace that links assessments to their sources and supports investigation of discrepancies; and Recursive Hierarchical Memory that retrieves and updates guidance from eligible task records and feedback.

The paper also introduces WxFlowBench, covering evidence-grounded analysis, nowcasting, multi-horizon forecasting, reporting, and weather services. Experiments include multiple language models, a vanilla comparison using the same model, tool and memory ablations, and failure analyses. The results provide evidence that an integrated agent workflow can improve meteorological task performance over the reported vanilla baseline. Evaluation uses historical replay; continuous operation with live data remains untested.

Critical strengths
  1. A valuable and comprehensive observation-to-service framework. The paper addresses the important problem of connecting weather data, forecasting tools, and downstream services. It combines an executable agent system with a benchmark covering five task groups. Experiments across several language models, tool ablations, and failure analyses provide a substantial foundation for studying weather agents.
  2. Clear rules for evidence use and service responsibilities. WxFlowBench distinguishes observations from forecasts and specifies information cutoffs. T4 also distinguishes potential impacts from observed losses and warning recommendations from official warning issuance. These rules provide concrete criteria for assessing whether an agent uses evidence correctly and communicates within its assigned role.
Critical weaknesses
  1. Human agreement validation of the T3/T4 evaluator is not reported. Factual Consistency (FC), Report Quality (RQ), and Groundedness (GRD) rely on LLM-assisted judgments of factual support, requirement satisfaction, and evidence support. The paper describes human review of reference reports and unresolved factual judgments, but does not report a comparison with independent human assessments of agent-generated reports. It remains unclear how accurately the evaluator scores these reports and whether scoring errors affect the reported improvements.
  2. Limited evidence linking individual components to improvements on C1–C3. The reported results show gains in overall T0 accuracy, T1/T2 forecast-source selection, and T3/T4 report quality with memory. However, the paper does not report targeted comparisons isolating the environment’s handling of heterogeneous data semantics (C1), the workspace’s contribution to conflict handling (C2), or recursive updating’s benefit over static experience for strategy selection (C3). The scope of w/o memory is also unclear. These gaps limit attribution of the observed gains to the proposed components.
Questions for authors
  1. Has the T3/T4 evaluator been checked against independent human assessments? If so, please report the sample size and selection method, evaluator expertise, and agreement results for FC, RQ, and GRD. Do disagreements or human corrections change the main comparative findings?
  2. Can you clarify the component-specific evidence for C1–C3? Beyond the aggregate results, are there targeted comparisons for data-semantics handling and evidence-conflict tasks that isolate the environment and workspace contributions? What exactly is removed in w/o memory, and is there a comparison of static versus recursively updated experience on forecast-source selection? If available, please summarize these results.
2 comments

Thank you for your detailed responses and additional experiments. The clarifications and new comparisons with ReAct and Zephyrus look good to me.

Inspired by the discussion from other reviewers, I have a additional question out of curiosity: How would MeteoSage compare with current SOTA general-purpose agent harnesses, such as Codex or Claude Code, if they were given the same meteorological data and tool access, with comparable execution budgets?

Given that these general-purpose harnesses already support sophisticated tool use, context management, and iterative reasoning, I am curious whether MeteoSage's specialized Evidence Workspace and Recursive Hierarchical Memory would provide additional benefits beyond them.

If feasible, I would appreciate a small-scale pilot experiment on a subset of WxFlowBench to explore this comparison. Such an experiment could help further clarify the incremental value of MeteoSage's specialized harness over existing general-purpose alternatives.

Thank you for your detailed responses and additional experiments. The clarifications and new comparisons with ReAct and Zephyrus look good to me.

Inspired by the discussion from other reviewers, I have a additional question out of curiosity: How would MeteoSage compare with current SOTA general-purpose agent harnesses, such as Codex or Claude Code, if they were given the same meteorological data and tool access, with comparable execution budgets?

Given that these general-purpose harnesses already support sophisticated tool use, context management, and iterative reasoning, I am curious whether MeteoSage's specialized Evidence Workspace and Recursive Hierarchical Memory would provide additional benefits beyond them.

If feasible, I would appreciate a small-scale pilot experiment on a subset of WxFlowBench to explore this comparison. Such an experiment could help further clarify the incremental value of MeteoSage's specialized harness over existing general-purpose alternatives.

Reviewer_2217OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agent framework plus five-track benchmark for operational weather workflow. MeteoSage: tool environment (data access, MetPy diagnostics, nowcasting via optical flow and SimVP, GraphCast and Pangu adapters, VLM visualization, code sandbox), evidence workspace with provenance and conflict protocol, hierarchical memory with experience distilled between tasks. WxFlowBench: T0 analysis 552 instances, T1 nowcasting, T2 forecasting 384, T3 reports 150, T4 services 150. Metrics: hit score, CRPS, Brier, FSS, LLM-judged FC and RQ, BLEU-1, completion, tool validity. Five backbones plus vanilla baseline (no harness!). Same-backbone gains big: T0 18.12 to 98.01, T2 completion 14.32 to 100, T3 RQ 6.96 to 52.53, T4 4.07 to 29.65.

Critical strengths
  • Yet I have not seen other benchmark covering full workflow (analysis, nowcasting, forecasting, reports, services) on real operational data.
  • Protocol documented OK: allocation constraints, group-preserving splits, frozen references independent of agent outputs.
  • Conflict taxonomy makes operational sense: station-grid differences, radar artifacts, ensemble spread treated as different mechanisms.
  • Ablations is reasonable
Critical weaknesses
  • Main comparison is straw man. Vanilla is same backbone with no tools, it cannot do T0/T2 by construction. 80-point gain shows environment works, not that agent reasons.
  • No statistics. No seeds, no intervals. Several contrasts inside noise.
  • FC and RQ judged by LLMs (updated after GPT-Astra release), no human agreement audit (looks like at least). Circularity risk when judge and evaluated families overlap.
  • Memory is headline contribution but weakly supported: ~6.5 RQ on T3, ~1 on T4, no learning curve for ``recursive'' claim.
  • Possible double-blind problem: github.com/useliz/MeteoSage -- while the account have only 1 public repo, so I am not 100% sure its a de-anonymization
Questions for authors
  1. ReAct with same tools on T0-T2: how much of gap survives?
  2. Per-track n plus seeds or intervals.
  3. Judge versions, human agreement rate on sample?
  4. Memory size, retrieval cost, performance vs experience version?
1 comment
Area_Chair_U87kOpenReview ↗

please engage

Dear reviewers,

Please join the discussion with the authors.

Best, AC

Reviewer_xkWoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MeteoSage, a meteorological agent system that supports weather analysis, forecasting, report generation, and public services through a unified data and tool environment, an evidence workspace that records the analytical process, and a memory mechanism that extracts lessons from past tasks. The authors also construct WxFlowBench to evaluate this workflow across multiple large language models and examine the system’s contributions and limitations through tool ablations and failure case analyses.

Critical strengths
  1. The paper addresses a practical problem. Meteorological analysis requires integrating data from multiple sources and resolving inconsistencies in time, units, and evidence. The paper organizes these steps into a fairly comprehensive workflow with practical value.
  2. WxFlowBench covers multiple stages, from analysis to service delivery, and represents a clear contribution. Beyond reporting aggregate scores, the paper examines failures such as models prematurely stopping exploration and incorrectly assessing forecast quality. These analyses help clarify the capability boundaries of meteorological agents.
Critical weaknesses
  1. My main concern is that the current comparisons do not sufficiently establish the framework’s added value. The primary baselines are Vanilla models without an agent framework, so much of the improvement may come from access to data and tools. The results therefore do not establish whether the proposed evidence management and memory mechanisms are more effective than those of a standard tool-using agent.
  2. The methodological contribution is also not sufficiently clear. The approach largely combines tool use, evidence logging, and experience retrieval. Although the paper includes tool ablations and experiments without memory, it does not adequately explain why experience needs to be updated recursively or what this improves over simple summaries of past tasks or a fixed experience repository. In particular, I would like to understand whether these designs actually help models resolve conflicts among meteorological evidence or develop better forecast selection strategies.
  3. There is also a gap between completing a task and producing a reliable report. Factual consistency remains relatively low for reporting and service tasks, despite high evidence support, and this discrepancy is not sufficiently explained. Moreover, the evaluation relies on replaying historical tasks, making it difficult to assess whether the system can consistently provide accurate and useful analyses in operational settings.
Questions for authors
  1. How much of MeteoSage’s advantage would remain if a standard tool-using agent were given the same data, tools, and tool-call budget? Could this comparison help disentangle the contributions of tool access and framework design?
  2. What specific errors does recursive memory reduce compared with a fixed experience repository or a one-time summary of past experience? Could tasks involving conflicting evidence or forecast selection demonstrate that it improves analysis and decision-making, rather than merely helping organize report content?
  3. How do the authors explain the gap between high evidence support and low factual consistency? Could meteorological experts review a small subset of complete reports to assess whether the key weather judgments, temporal and geographical information, and service recommendations are accurate and useful?
1 comment
Area_Chair_U87kOpenReview ↗

please engage

Dear reviewers,

Please join the discussion with the authors.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.