- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how AI agents can combine heterogeneous meteorological evidence and forecasting tools to produce traceable analyses, forecasts, and service outputs. It addresses three challenges: differences in data semantics, conflicting evidence, and forecasting strategies that depend on the task context. MeteoSage combines three components: a Meteorological Agentic Environment that preserves data definitions and provides scientific tools; an Event-Centered Evidence Workspace that links assessments to their sources and supports investigation of discrepancies; and Recursive Hierarchical Memory that retrieves and updates guidance from eligible task records and feedback.
The paper also introduces WxFlowBench, covering evidence-grounded analysis, nowcasting, multi-horizon forecasting, reporting, and weather services. Experiments include multiple language models, a vanilla comparison using the same model, tool and memory ablations, and failure analyses. The results provide evidence that an integrated agent workflow can improve meteorological task performance over the reported vanilla baseline. Evaluation uses historical replay; continuous operation with live data remains untested.
Critical strengths
- A valuable and comprehensive observation-to-service framework. The paper addresses the important problem of connecting weather data, forecasting tools, and downstream services. It combines an executable agent system with a benchmark covering five task groups. Experiments across several language models, tool ablations, and failure analyses provide a substantial foundation for studying weather agents.
- Clear rules for evidence use and service responsibilities. WxFlowBench distinguishes observations from forecasts and specifies information cutoffs. T4 also distinguishes potential impacts from observed losses and warning recommendations from official warning issuance. These rules provide concrete criteria for assessing whether an agent uses evidence correctly and communicates within its assigned role.
Critical weaknesses
- Human agreement validation of the T3/T4 evaluator is not reported. Factual Consistency (FC), Report Quality (RQ), and Groundedness (GRD) rely on LLM-assisted judgments of factual support, requirement satisfaction, and evidence support. The paper describes human review of reference reports and unresolved factual judgments, but does not report a comparison with independent human assessments of agent-generated reports. It remains unclear how accurately the evaluator scores these reports and whether scoring errors affect the reported improvements.
- Limited evidence linking individual components to improvements on C1–C3. The reported results show gains in overall T0 accuracy, T1/T2 forecast-source selection, and T3/T4 report quality with memory. However, the paper does not report targeted comparisons isolating the environment’s handling of heterogeneous data semantics (C1), the workspace’s contribution to conflict handling (C2), or recursive updating’s benefit over static experience for strategy selection (C3). The scope of
w/o memoryis also unclear. These gaps limit attribution of the observed gains to the proposed components.
Questions for authors
- Has the T3/T4 evaluator been checked against independent human assessments? If so, please report the sample size and selection method, evaluator expertise, and agreement results for FC, RQ, and GRD. Do disagreements or human corrections change the main comparative findings?
- Can you clarify the component-specific evidence for C1–C3? Beyond the aggregate results, are there targeted comparisons for data-semantics handling and evidence-conflict tasks that isolate the environment and workspace contributions? What exactly is removed in
w/o memory, and is there a comparison of static versus recursively updated experience on forecast-source selection? If available, please summarize these results.
2 comments
Thank you for your detailed responses and additional experiments. The clarifications and new comparisons with ReAct and Zephyrus look good to me.
Inspired by the discussion from other reviewers, I have a additional question out of curiosity: How would MeteoSage compare with current SOTA general-purpose agent harnesses, such as Codex or Claude Code, if they were given the same meteorological data and tool access, with comparable execution budgets?
Given that these general-purpose harnesses already support sophisticated tool use, context management, and iterative reasoning, I am curious whether MeteoSage's specialized Evidence Workspace and Recursive Hierarchical Memory would provide additional benefits beyond them.
If feasible, I would appreciate a small-scale pilot experiment on a subset of WxFlowBench to explore this comparison. Such an experiment could help further clarify the incremental value of MeteoSage's specialized harness over existing general-purpose alternatives.
Thank you for your detailed responses and additional experiments. The clarifications and new comparisons with ReAct and Zephyrus look good to me.
Inspired by the discussion from other reviewers, I have a additional question out of curiosity: How would MeteoSage compare with current SOTA general-purpose agent harnesses, such as Codex or Claude Code, if they were given the same meteorological data and tool access, with comparable execution budgets?
Given that these general-purpose harnesses already support sophisticated tool use, context management, and iterative reasoning, I am curious whether MeteoSage's specialized Evidence Workspace and Recursive Hierarchical Memory would provide additional benefits beyond them.
If feasible, I would appreciate a small-scale pilot experiment on a subset of WxFlowBench to explore this comparison. Such an experiment could help further clarify the incremental value of MeteoSage's specialized harness over existing general-purpose alternatives.