- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces EarthVerse, a benchmark for evaluating scientific agents in Earth-system and natural-hazard analysis. It aims to assess agents' capabilities in multi-source evidence retrieval, information integration, quantitative computation, and scientific reasoning. The authors evaluate 25 models and agent systems and conduct controlled experiments on reasoning, tool use, and evidence management. The benchmark addresses a potentially valuable research direction. However, the paper does not sufficiently articulate its scientific motivation, evaluation scope, benchmark construction methodology, or core contributions, making it difficult to fully assess its novelty and significance.
Critical strengths
- Potential practical value. The paper extends scientific agent evaluation beyond question answering with predefined evidence to more complex tasks requiring autonomous evidence retrieval and scientific computation.
- Relatively large evaluation scale and diverse experimental settings. The benchmark covers multiple natural-hazard categories and evaluates 25 models and agent systems.
Critical weaknesses
W1: Unclear Research Positioning and Insufficiently Articulated Scientific Contributions
- Ambiguous research scope. The title places "Dynamic Earth Systems" and "Natural Hazards" alongside each other, but they represent different conceptual levels. The benchmark primarily focuses on retrospective natural-hazard investigations, which do not adequately represent the broader scope of dynamic Earth-system research. Moreover, it remains unclear how fixed historical event packages meaningfully capture or evaluate the "dynamic" aspect.
- Insufficient research motivation. The Abstract and Introduction do not clearly establish the central research problem, target scenarios, necessity of the benchmark, or scientific significance. In particular, the paper does not sufficiently explain which critical capabilities EarthVerse evaluates or which limitations of existing benchmarks it addresses.
- Limited justification of novelty and contributions. The proposed capability categories are broadly defined, with insufficient clarification of their precise meanings and distinctions. The paper does not adequately explain how these capabilities differ from general tool use, multi-step reasoning, or those already assessed by existing scientific agent benchmarks.
- Lack of systematic related work analysis. The Related Work section largely lists existing methods and benchmarks without providing an in-depth comparison. Consequently, the distinctive contributions of EarthVerse remain insufficiently established. W2: Insufficient Details on Benchmark
- Limited transparency in event selection and data construction. Although the appendix lists data-source categories, the original collection channels and data acquisition procedures for the 199 events are not systematically documented.
- Insufficient analysis of dataset coverage and representativeness. The paper does not adequately examine the distribution of hazard categories, geographical regions, temporal coverage, and data modalities, or establish whether the selected events and tasks are sufficiently representative.
- Unclear mapping between tasks and evaluated capabilities. The benchmark lacks sufficiently detailed task construction criteria, justification for capability annotations, and representative examples, making it difficult to determine whether the tasks effectively assess the claimed scientific capabilities.
- Insufficient validation of evaluation metrics. Further justification and empirical validation are needed to establish whether the proposed metrics accurately reflect the quality and reliability of scientific analysis. W3: Significant Issues with Organization and Presentation
- Lack of a coherent narrative. The manuscript contains numerous high-level descriptions but lacks a clear progression from scientific motivation and benchmark design to experimental validation and research contributions. The central idea is not sufficiently emphasized.
- Poor organization of methodological details. Important information on data construction, task design, and evaluation protocols is scattered across the main text and appendices. The main paper remains overly high-level, making it difficult to develop a comprehensive understanding of the benchmark.
- Superficial Abstract and Introduction. The research significance and core contributions are not articulated with sufficient depth. The writing frequently relies on generic and repetitive statements, with some passages exhibiting formulaic, AI-like phrasing that weakens the logical flow and precision of the scientific presentation.
- Figures require improvement. The numerical annotations and statistical meanings in Figure 1(a) are not sufficiently clear. Figure 1(b) should clarify the model selection criteria and consider incorporating more recent representative frontier models.
Questions for authors
Q1: Research Positioning and Evaluation Scope Why are "Dynamic Earth Systems" and "Natural Hazards" presented as parallel concepts? How does a benchmark based on fixed historical event packages capture and evaluate the "dynamic" characteristics claimed in the title? Q2: Event Selection and Data Construction What are the original sources, selection criteria, and multi-source data collection procedures for the events? How representative is the dataset across hazard categories, geographical regions, and time periods? Could the authors clarify the specific acquisition pipelines used to download and collect data from different sources? Q3: Novelty and Distinctive Contributions Compared with Earth-Agent, DORA, and GeoMMBench, what is the most significant and distinctive contribution of EarthVerse?
11 comments
Response to Reviewer Uo12 — Q2(a): Event selection, original sources, and acquisition pipeline
Question: What are the event-selection criteria, original sources, and multi-source acquisition procedures?
Thank you for asking about the sources and acquisition pipeline. Section 3.4 and Figure 5 describe event verification, package assembly, task authoring, executable validation, and release review. Appendix C.3 provides the source-family inventory, acquisition interfaces, and scientific uses.
Event selection. We select documented disasters and extremes with scientific relevance and observations that support reproducible multi-source investigation. Experts cross-check event identity, timing, and footprint against catalogs and authoritative reports. Event anchors include GDACS, NASA EONET, ReliefWeb, meteorological agencies, and disaster-response archives.
Data acquisition and provenance. Observations are collected through public APIs and archives, including reanalysis and precipitation products, historical weather services, satellite and STAC interfaces, ocean and climate-index products, population grids, and OpenStreetMap. The event context determines temporal windows and spatial queries.
Package manifests record source URLs, acquisition status, and the corresponding local files. Aggregates and change products retain their upstream provenance.
Scientific task construction. Earth-science experts define the question, evidence requirements, reference calculations, answer units, and rubric. Supporting reviewers check sources, metadata, and calculations. Agent-assisted replay identifies discrepancies for human correction, and deterministic computation verifies that the evidence regenerates the reference targets. This connects source provenance, scientific interpretation, and executable evaluation within each task.
Response to Reviewer Uo12 — Q2(b): Hazard coverage of the release and supplementary evaluation
Question: How is the dataset distributed across hazard categories?
Thank you for asking how event selection is distributed across hazards. Section 3’s published taxonomy spans meteorological, hydrological, geophysical, cryospheric, ecological, and compound events.
Hazard composition (%). The table places the full release alongside the frozen supplementary cohort used for the newer-model experiments. The supplementary distribution broadly follows the release taxonomy, and all compared models use the same task list.
\hline \text{Hazard family} & \text{Release share} & \text{Supplement share} \\\\ \hline \text{Air pollution / smoke / heat} & 4.94 & 5.00 \\\\ \text{Atmospheric river / monsoon} & 3.95 & 4.00 \\\\ \text{Climate teleconnection} & 5.19 & 5.00 \\\\ \text{Cold wave / winter storm} & 2.96 & 3.00 \\\\ \text{Compound / cascading} & 5.43 & 6.00 \\\\ \text{Cryosphere} & 4.94 & 5.00 \\\\ \text{Drought / agriculture} & 0.99 & 1.00 \\\\ \text{Dust / sandstorm} & 4.44 & 4.00 \\\\ \text{Earthquake / tsunami} & 6.17 & 6.00 \\\\ \text{Extratropical / convection} & 4.44 & 4.00 \\\\ \text{Extreme rain / pluvial flood} & 13.09 & 13.00 \\\\ \text{Heat-wave / drought} & 0.49 & 0.00 \\\\ \text{Heat-wave / urban heat} & 7.16 & 8.00 \\\\ \text{Landslide} & 4.94 & 5.00 \\\\ \text{Marine heat-wave / coastal} & 3.21 & 3.00 \\\\ \text{River / coastal flood} & 8.40 & 9.00 \\\\ \text{Tropical cyclone} & 6.67 & 7.00 \\\\ \text{Volcano / ash / lahar} & 3.46 & 3.00 \\\\ \text{Wildfire / burn / smoke} & 9.14 & 9.00 \\\\ \hline \end{array}$$
Response to Reviewer Uo12 — Q1: Research positioning and dynamic Earth-system processes
Question: How do fixed event packages capture dynamic Earth-system processes, and how does this relate to natural hazards?
Thank you for recognizing the practical value of EarthVerse in evaluating evidence retrieval and scientific computation, together with the scale of the evaluation and its diverse experimental settings.
Earth-system processes and hazard events. The title connects evolving physical processes with the natural-hazard events through which we examine them. EarthVerse evaluates how an agent reconstructs these processes from heterogeneous observations and produces an evidence-supported event account.
The dynamic information resides in the observations. An event archive contains records from different stages, temporal resolutions, and spatial supports. Appendix F’s heat-wave example requires matching hourly observations and daily aggregates to the event window, then computing temperature and humidity statistics on compatible support. The wildfire and flood cases in Appendix G similarly connect timing, footprints, measurements, and physical interpretation.
Updated evaluation of process-related capabilities. We continued these investigations with newer models. The results below describe GPT-6.1 Sol, with Strict@95 reported as a percentage.
\hline \text{Capability} & \text{Core} & \text{Strict@95} \\\\ \hline \text{Spatiotemporal process} & 92.03 & 75.00 \\\\ \text{Physical mechanism} & 86.37 & 60.00 \\\\ \text{Causal chain} & 83.20 & 48.21 \\\\ \hline \end{array}$$ Fixing the evidence supports reproducibility while preserving the event evolution that the agent reconstructs. These task slices connect evaluation directly to temporal change, physical mechanisms, and causal reasoning.