acceptodds
Withdrawn from ICLR 2027

EXConan: A Global Benchmark and Evidence-Grounded Multi-Agent Framework for Extreme Weather Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While recent large language models have demonstrated remarkable general reasoning capabilities, their potential in Earth system sciences remains bottlenecked by the complexity of meteorological tasks. Existing evaluations primarily target numerical forecasting, leaving a critical gap in assessing LLMs for evidence-grounded extreme weather reasoning that requires multi-source evidence interpretation and domain-specific knowledge. To address this, we introduce EXConan-Bench, a comprehensive global benchmark comprising 8,498 expert-curated instances. By integrating multi-source event records with ERA5 meteorological reanalysis data, the benchmark covers eight extreme weather hazards and normal conditions, challenging models to identify events using structured atmospheric evidence and interpretable rationales. Furthermore, we propose \conans EXConan, an evidence-grounded multi-agent reasoning framework designed to overcome the limitations of direct prompting. EXConan introduces an agentic paradigm that meticulously coordinates specialized meteorological experts, evidence-based candidate evaluation, and counterfactual verification to ensure reliable decision-making. Extensive experiments across representative closed-source and open-source LLMs reveal the significant limitations of current baselines in meteorological reasoning. Remarkably, our EXConan framework effectively mitigates these shortcomings; it improves Qwen3-30B from 49.33% to 54.63%, with consistent gains in Macro Precision, Weighted-F1, and Kappa. Ultimately, EXConan establishes a scientifically grounded paradigm for reliable weather intelligence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Extreme Weather Reasoning, Multi-Agent System

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kLm6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EXConan-Bench, a benchmark for evaluating how well large language models (LLMs) recognize extreme weather events globally from ERA5-derived atmospheric evidence. It also proposes EXConan, a multi-agent framework that combines meteorological specialist agents, comparison of evidence across candidate events and counterfactual verification. The experimental results show EXConan consistently improves performance across backbones.

Critical strengths

The paper is well-presented and easy to follow.

The proposed benchmark has comprehensive coverage across multi-decadal and multi-hazard scenarios under a consistent 24-hour pre-target cutoff and a class-independent report format. This enables comparable evaluation across regions, time periods and hazards, which existing weather benchmarks lack.

The proposed EXConan multi-agent framework materializes the evidences observed from the benchmark, and has promising improvement regarding experimental results across multiple models.

Critical weaknesses

A large portion (~30%) of the benchmark comes from "legacy benchmark", which is an unnamed source whose labels have no documented origin or validation. This undermines the central contribution of “8,498 expert-curated instances” as a benchmark.

For the result breakdown in table S15, the ExConan framework seems to lower the "ERA5 heatwave threshold mining" performance substantially (-26.19). Could the authors provide source-stratified results on each model benchmarked (or average stratified results) to see if this is a consistent behavior? If so, this is likely a limitation of the proposed framework.

ExConan framework seems to use ~7 times more tokens than the base model on the EXConan Bench-950 tasks, while still underperforming the closed source base model (i.e. Gemini 3.6 Flash). This also excludes the Aurora model's GPU inference cost. Benchmarking ExConan framework on closed-source model is a tempting experiment to add in order to show the ceiling of improvement of the framework to justify the extra token costs.

Questions for authors

Please refer to the strengths and weaknesses.

Reviewer_fEd8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EXConan-Bench, a global benchmark containing 8,498 region–target-time instances across eight extreme-weather categories and a non-extreme control. Models select one of four candidate categories using structured ERA5 reports with a shared pre-target evidence cutoff. The paper also proposes a multi-agent framework combining meteorological specialist roles, evidence comparison, arbitration, and conditional review, with supplementary Aurora forecasts in selected cases. The main text reports an accuracy improvement from 49.33% to 54.63% for Qwen3-30B. Controlled comparisons and ablations examine the framework's components and input dependence.

Critical strengths
  1. The benchmark has broad coverage and a clear input protocol. The instances cover multiple hazards across globally distributed regions and an extended historical period. The common T−24h evidence cutoff and class-independent report template help constrain the available information and reduce label cues introduced by category-specific report formats, providing a common basis for cross-hazard comparisons.

  2. The reasoning process is inspectable, and the evaluation includes useful diagnostics. The framework requires references to numbered evidence and distinguishes supporting from contradicting information. Same-backbone comparisons, multi-sample and role-voting baselines, component ablations, and experiments with three generation seeds help readers inspect decision grounds and the sources of performance changes.

Critical weaknesses
  1. Inconsistencies in the core results prevent reliable assessment of the main performance claims. Table 2 reports 38.61% accuracy for the Qwen3-4B framework, whereas Table S7 reports a final result of 39.96%. Direct Phi-4 accuracy is 27.73% in Table 2 but 47.21% in Table S5; this discrepancy can even change whether the framework appears to improve on its backbone. These results are all described as evaluations on 8,498 instances. The configurations, run versions, and scoring denominators must be reconciled before the claimed performance advantages can be verified.

  2. Evidence-grounded meteorological mechanism reasoning is insufficiently validated. In the 950-instance Qwen3-4B direct-inference ablation, location/time metadata and candidate options alone achieve 32.00% accuracy, compared with 31.68% for the complete input. Corresponding ablations are not reported for the full framework, leaving its reliance on meteorological evidence unverified. Although the reports contain sector summaries and temporal evolution, winds are reduced to speed without directional information, and 500 hPa geopotential height lacks anomalies and the full large-scale spatial structure. These limitations constrain the mechanisms that the supplied evidence can support. The experiments do not sufficiently establish whether specialist roles provide scientifically grounded interpretations beyond the benefits of role-conditioned prompt diversity, which directly affects the paper's central positioning.

Questions for authors
  1. Why do the Qwen3-4B and Phi-4 results differ between Table 2 and Tables S5–S7? What configurations, run versions, and denominators do they represent, and which results should underpin the final conclusions?

  2. How does the complete framework perform when the weather report is removed? Are input-ablation results available that establish its dependence on meteorological evidence beyond location/time metadata and candidate priors?

  3. Given the absence of wind direction, climatological anomalies, and complete large-scale fields, what available evidence supports the specialists' circulation or mechanism explanations? Are scientific verification results available to substantiate these explanations and the role specialization?

Reviewer_Cu3wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Benchmark plus framework. EXConan-Bench: 8498 instances, 2000-2023, eight hazard classes plus no-extreme control (18.7%), built from EM-DAT, SPEI drought mining, ERA5 sampling, model-assisted expansion. Each instance has structured ERA5 report (48h evolution + 7 and 30 day context), strict cutoff at T-24h, four-choice decision. QA is automated checks, GPT-assisted checks, staged human review.

EXConan: three specialists (synoptic pattern, hazard severity, temporal evolution) score all candidates with evidence citations, evidence competition with contradiction penalty (lambda 0.15) plus LLM arbiter, deterministic stress test flags fragile decisions for one counterfactual review, Aurora forecast as extra evidence for unanimous no-extreme cases.

Results: 15 direct configurations, framework on four open backbones. Qwen3-30B 49.33 to 54.63 accuracy, weighted F1 50.27 to 53.43, kappa 0.4214 to 0.4915. On dev subset (Qwen3-4B): 36.95 vs 31.68 direct, 30.74 self-consistency, 33.58 role voting, 4.73 calls per case. All component ablations negative (arbiter -6.21, tools -4.63, generalist -4.42, competition -2.42)

Intersting benchmark; however, I have some concerns that need to be addressed

Critical strengths
  • Benchmark fills real gap, Table 1 shows fragmentation of existing work. Global, 24 years, multi-hazard, one cutoff, one protocol.
  • Construction careful: category-neutral template so input format cannot leak label, leakage checks, dedup documented, class counts add up (I checked, sum is 8498).
  • Framework comes from real error analysis, controls right (matched call budgets). They admit role voting beats them on macro precision, credit for this.
  • Best statistics I have seen in this kind of paper: exact McNemar, bootstrap, Holm, three seeds, honest disclosure that lambda and gates are unswept heuristics.
  • Dev subset openly included in full benchmark numbers, no pretending it is held out
Critical weaknesses
  • Gains modest. Best framework result 54.63 still below Gemini 3.6 Flash direct 56.53. Framework never tried on frontier backbone, cost never quantified. Calls are not dollars.
  • No memorization check. Events are real historical disasters, model can recall them from pretraining instead of reading evidence. Cutoff stops input leakage only. This is my main concern.
  • LLM-assisted labeling (Qwen2.5-72B) plus GPT-assisted QA is mild circularity for an LLM benchmark. No inter-annotator agreement reported.
  • Full benchmark looks single run per config, region and month slices small.
  • Aurora trigger narrow, role in main config unclear.
  • Four-choice with answer always present bounds difficulty, no open-ended variant
Questions for authors
  1. Framework on one frontier backbone, even dev-950 only. Gains persist or specialists saturate?
  2. Perturbed or blinded events: how much of performance is memorization?
  3. Inter-annotator agreement, escalation rate?
  4. Stress test flag rate, overturn vs keep, split by initial no-extreme?
  5. Sensitivity of lambda 0.15 and confidence gates?
  6. How are overlapping hazards labeled, how common?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.