- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EXConan-Bench, a benchmark for evaluating how well large language models (LLMs) recognize extreme weather events globally from ERA5-derived atmospheric evidence. It also proposes EXConan, a multi-agent framework that combines meteorological specialist agents, comparison of evidence across candidate events and counterfactual verification. The experimental results show EXConan consistently improves performance across backbones.
Critical strengths
The paper is well-presented and easy to follow.
The proposed benchmark has comprehensive coverage across multi-decadal and multi-hazard scenarios under a consistent 24-hour pre-target cutoff and a class-independent report format. This enables comparable evaluation across regions, time periods and hazards, which existing weather benchmarks lack.
The proposed EXConan multi-agent framework materializes the evidences observed from the benchmark, and has promising improvement regarding experimental results across multiple models.
Critical weaknesses
A large portion (~30%) of the benchmark comes from "legacy benchmark", which is an unnamed source whose labels have no documented origin or validation. This undermines the central contribution of “8,498 expert-curated instances” as a benchmark.
For the result breakdown in table S15, the ExConan framework seems to lower the "ERA5 heatwave threshold mining" performance substantially (-26.19). Could the authors provide source-stratified results on each model benchmarked (or average stratified results) to see if this is a consistent behavior? If so, this is likely a limitation of the proposed framework.
ExConan framework seems to use ~7 times more tokens than the base model on the EXConan Bench-950 tasks, while still underperforming the closed source base model (i.e. Gemini 3.6 Flash). This also excludes the Aurora model's GPU inference cost. Benchmarking ExConan framework on closed-source model is a tempting experiment to add in order to show the ceiling of improvement of the framework to justify the extra token costs.
Questions for authors
Please refer to the strengths and weaknesses.