acceptodds
Under review as a conference paper at ICLR 2027

EarthVerse: Benchmarking Scientific Agents across Dynamic Earth Systems and Natural Hazards

12 views

Abstract

Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems. Our code is available at [code](https://anonymous.4open.science/r/Earth-Verse-7618/).

Primary area:
datasets and benchmarks
Keywords:
scientific agents, Earth system science, multi-source reasoning, tool use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Uo12OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces EarthVerse, a benchmark for evaluating scientific agents in Earth-system and natural-hazard analysis. It aims to assess agents' capabilities in multi-source evidence retrieval, information integration, quantitative computation, and scientific reasoning. The authors evaluate 25 models and agent systems and conduct controlled experiments on reasoning, tool use, and evidence management. The benchmark addresses a potentially valuable research direction. However, the paper does not sufficiently articulate its scientific motivation, evaluation scope, benchmark construction methodology, or core contributions, making it difficult to fully assess its novelty and significance.

Critical strengths
  1. Potential practical value. The paper extends scientific agent evaluation beyond question answering with predefined evidence to more complex tasks requiring autonomous evidence retrieval and scientific computation.
  2. Relatively large evaluation scale and diverse experimental settings. The benchmark covers multiple natural-hazard categories and evaluates 25 models and agent systems.
Critical weaknesses

W1: Unclear Research Positioning and Insufficiently Articulated Scientific Contributions

  1. Ambiguous research scope. The title places "Dynamic Earth Systems" and "Natural Hazards" alongside each other, but they represent different conceptual levels. The benchmark primarily focuses on retrospective natural-hazard investigations, which do not adequately represent the broader scope of dynamic Earth-system research. Moreover, it remains unclear how fixed historical event packages meaningfully capture or evaluate the "dynamic" aspect.
  2. Insufficient research motivation. The Abstract and Introduction do not clearly establish the central research problem, target scenarios, necessity of the benchmark, or scientific significance. In particular, the paper does not sufficiently explain which critical capabilities EarthVerse evaluates or which limitations of existing benchmarks it addresses.
  3. Limited justification of novelty and contributions. The proposed capability categories are broadly defined, with insufficient clarification of their precise meanings and distinctions. The paper does not adequately explain how these capabilities differ from general tool use, multi-step reasoning, or those already assessed by existing scientific agent benchmarks.
  4. Lack of systematic related work analysis. The Related Work section largely lists existing methods and benchmarks without providing an in-depth comparison. Consequently, the distinctive contributions of EarthVerse remain insufficiently established. W2: Insufficient Details on Benchmark
  5. Limited transparency in event selection and data construction. Although the appendix lists data-source categories, the original collection channels and data acquisition procedures for the 199 events are not systematically documented.
  6. Insufficient analysis of dataset coverage and representativeness. The paper does not adequately examine the distribution of hazard categories, geographical regions, temporal coverage, and data modalities, or establish whether the selected events and tasks are sufficiently representative.
  7. Unclear mapping between tasks and evaluated capabilities. The benchmark lacks sufficiently detailed task construction criteria, justification for capability annotations, and representative examples, making it difficult to determine whether the tasks effectively assess the claimed scientific capabilities.
  8. Insufficient validation of evaluation metrics. Further justification and empirical validation are needed to establish whether the proposed metrics accurately reflect the quality and reliability of scientific analysis. W3: Significant Issues with Organization and Presentation
  9. Lack of a coherent narrative. The manuscript contains numerous high-level descriptions but lacks a clear progression from scientific motivation and benchmark design to experimental validation and research contributions. The central idea is not sufficiently emphasized.
  10. Poor organization of methodological details. Important information on data construction, task design, and evaluation protocols is scattered across the main text and appendices. The main paper remains overly high-level, making it difficult to develop a comprehensive understanding of the benchmark.
  11. Superficial Abstract and Introduction. The research significance and core contributions are not articulated with sufficient depth. The writing frequently relies on generic and repetitive statements, with some passages exhibiting formulaic, AI-like phrasing that weakens the logical flow and precision of the scientific presentation.
  12. Figures require improvement. The numerical annotations and statistical meanings in Figure 1(a) are not sufficiently clear. Figure 1(b) should clarify the model selection criteria and consider incorporating more recent representative frontier models.
Questions for authors

Q1: Research Positioning and Evaluation Scope Why are "Dynamic Earth Systems" and "Natural Hazards" presented as parallel concepts? How does a benchmark based on fixed historical event packages capture and evaluate the "dynamic" characteristics claimed in the title? Q2: Event Selection and Data Construction What are the original sources, selection criteria, and multi-source data collection procedures for the events? How representative is the dataset across hazard categories, geographical regions, and time periods? Could the authors clarify the specific acquisition pipelines used to download and collect data from different sources? Q3: Novelty and Distinctive Contributions Compared with Earth-Agent, DORA, and GeoMMBench, what is the most significant and distinctive contribution of EarthVerse?

11 comments
AuthorsOpenReview ↗

Response to Reviewer Uo12 — Q1: Research positioning and dynamic Earth-system processes

Question: How do fixed event packages capture dynamic Earth-system processes, and how does this relate to natural hazards?

Thank you for recognizing the practical value of EarthVerse in evaluating evidence retrieval and scientific computation, together with the scale of the evaluation and its diverse experimental settings.

Earth-system processes and hazard events. The title connects evolving physical processes with the natural-hazard events through which we examine them. EarthVerse evaluates how an agent reconstructs these processes from heterogeneous observations and produces an evidence-supported event account.

The dynamic information resides in the observations. An event archive contains records from different stages, temporal resolutions, and spatial supports. Appendix F’s heat-wave example requires matching hourly observations and daily aggregates to the event window, then computing temperature and humidity statistics on compatible support. The wildfire and flood cases in Appendix G similarly connect timing, footprints, measurements, and physical interpretation.

Updated evaluation of process-related capabilities. We continued these investigations with newer models. The results below describe GPT-6.1 Sol, with Strict@95 reported as a percentage.

\hline \text{Capability} & \text{Core} & \text{Strict@95} \\\\ \hline \text{Spatiotemporal process} & 92.03 & 75.00 \\\\ \text{Physical mechanism} & 86.37 & 60.00 \\\\ \text{Causal chain} & 83.20 & 48.21 \\\\ \hline \end{array}$$ Fixing the evidence supports reproducibility while preserving the event evolution that the agent reconstructs. These task slices connect evaluation directly to temporal change, physical mechanisms, and causal reasoning.
AuthorsOpenReview ↗

Response to Reviewer Uo12 — Q2(a): Event selection, original sources, and acquisition pipeline

Question: What are the event-selection criteria, original sources, and multi-source acquisition procedures?

Thank you for asking about the sources and acquisition pipeline. Section 3.4 and Figure 5 describe event verification, package assembly, task authoring, executable validation, and release review. Appendix C.3 provides the source-family inventory, acquisition interfaces, and scientific uses.

Event selection. We select documented disasters and extremes with scientific relevance and observations that support reproducible multi-source investigation. Experts cross-check event identity, timing, and footprint against catalogs and authoritative reports. Event anchors include GDACS, NASA EONET, ReliefWeb, meteorological agencies, and disaster-response archives.

Data acquisition and provenance. Observations are collected through public APIs and archives, including reanalysis and precipitation products, historical weather services, satellite and STAC interfaces, ocean and climate-index products, population grids, and OpenStreetMap. The event context determines temporal windows and spatial queries.

Package manifests record source URLs, acquisition status, and the corresponding local files. Aggregates and change products retain their upstream provenance.

Scientific task construction. Earth-science experts define the question, evidence requirements, reference calculations, answer units, and rubric. Supporting reviewers check sources, metadata, and calculations. Agent-assisted replay identifies discrepancies for human correction, and deterministic computation verifies that the evidence regenerates the reference targets. This connects source provenance, scientific interpretation, and executable evaluation within each task.

AuthorsOpenReview ↗

Response to Reviewer Uo12 — Q2(b): Hazard coverage of the release and supplementary evaluation

Question: How is the dataset distributed across hazard categories?

Thank you for asking how event selection is distributed across hazards. Section 3’s published taxonomy spans meteorological, hydrological, geophysical, cryospheric, ecological, and compound events.

Hazard composition (%). The table places the full release alongside the frozen supplementary cohort used for the newer-model experiments. The supplementary distribution broadly follows the release taxonomy, and all compared models use the same task list.

\hline \text{Hazard family} & \text{Release share} & \text{Supplement share} \\\\ \hline \text{Air pollution / smoke / heat} & 4.94 & 5.00 \\\\ \text{Atmospheric river / monsoon} & 3.95 & 4.00 \\\\ \text{Climate teleconnection} & 5.19 & 5.00 \\\\ \text{Cold wave / winter storm} & 2.96 & 3.00 \\\\ \text{Compound / cascading} & 5.43 & 6.00 \\\\ \text{Cryosphere} & 4.94 & 5.00 \\\\ \text{Drought / agriculture} & 0.99 & 1.00 \\\\ \text{Dust / sandstorm} & 4.44 & 4.00 \\\\ \text{Earthquake / tsunami} & 6.17 & 6.00 \\\\ \text{Extratropical / convection} & 4.44 & 4.00 \\\\ \text{Extreme rain / pluvial flood} & 13.09 & 13.00 \\\\ \text{Heat-wave / drought} & 0.49 & 0.00 \\\\ \text{Heat-wave / urban heat} & 7.16 & 8.00 \\\\ \text{Landslide} & 4.94 & 5.00 \\\\ \text{Marine heat-wave / coastal} & 3.21 & 3.00 \\\\ \text{River / coastal flood} & 8.40 & 9.00 \\\\ \text{Tropical cyclone} & 6.67 & 7.00 \\\\ \text{Volcano / ash / lahar} & 3.46 & 3.00 \\\\ \text{Wildfire / burn / smoke} & 9.14 & 9.00 \\\\ \hline \end{array}$$
Reviewer_t7JiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EarthVerse, a benchmark for scientific agents comprising 405 investigations across 199 events and 19 natural-hazard families. Agents inspect heterogeneous evidence packages, select and reconcile sources, perform calculations, and produce scientifically supported conclusions. Evaluation combines answer correctness with process assessment and uses Strict@95 to measure near-complete task completion. The paper compares 25 model–controller systems and conducts controlled GPT-5.5 experiments on reasoning effort, evidence access, and final review. The results show a gap between average answer-unit accuracy and near-complete investigations, while evidence-localization assistance produces substantial improvements in the tested setting.

Critical strengths
  1. The benchmark addresses an interesting challenge in evaluating scientific agents. Sections 3.1–3.3 require agents to select compatible evidence, reconcile different spatial and temporal scales, perform calculations, and preserve provenance. The collection spans 405 investigations and 19 hazard families, providing substantial coverage of these challenges. Separating answer-unit completion from near-complete task performance also provides a useful perspective on how partial progress translates into a complete scientific investigation.

  2. The controlled experiments provide useful evidence about the role of evidence access. Section 5.2 and Table 3(b) compare answer-free assistance on matched tasks. Relevant-file localization improves Core by 14.72 points, with a 95% confidence interval of [5.48, 24.69], while an evidence map improves it by 11.34 points, with an interval of [2.34, 21.25]. The paired design and reported uncertainty strengthen the evidence that locating relevant observations is an important bottleneck for the evaluated GPT-5.5 configuration. These analyses make the benchmark informative beyond its leaderboard.

Critical weaknesses
  1. The validity of the automated scoring is not sufficiently established. Sections 4-4.1 use a fixed GPT-4.1 judge to assess answers and research traces, with the process score contributing half of Combined Core. The paper describes substantial expert review of task construction, but does not report quantitative agreement between the judge and independent experts on actual agent outputs. The additional LLM pairwise comparison provides a useful consistency check, but does not establish that the grading reliably recognizes valid alternative analyses or consequential scientific errors. Because the leaderboard and reliability conclusions depend on these judgments, this limits confidence in the benchmark’s central measurements.

  2. The currently available materials do not support independent verification of the full benchmark. Section 3.2 describes executable ground truth for every task, but the Reproducibility Statement reports that the released evidence packages cover only 10 events and 22 of the 405 tasks.

Questions for authors
  1. How closely do the automated answer-unit and process scores agree with independent expert ratings on a representative sample of actual agent outputs, including scientifically valid alternative analyses?

  2. Can the authors provide anonymous access during review to the complete event packages, ground-truth computation scripts, and representative scored trajectories so that the full benchmark can be independently checked?

8 comments
AuthorsOpenReview ↗

Response to Reviewer t7Ji — Q1: Independent expert validation and alternative analyses

Question: How closely do automated scores agree with independent experts on actual outputs, including valid alternative analyses?

Thank you for recognizing the value of multi-source scientific investigation, the breadth of EarthVerse’s hazard coverage, and the distinction between partial progress and near-complete investigations. We particularly appreciate your positive assessment of the paired evidence-localization experiments.

Independent expert evaluation. Earth-science experts independently evaluated actual answers and research traces, with system identities and GPT-4.1 scores hidden. The material covered different model families, hazards, performance levels, and scientifically valid alternative analyses. Expert–judge agreement closely tracks expert–expert agreement, including process scores. Agreement columns are percentages; MAE is measured on the process rubric’s scale.

\hline \text{Scoring pair} & \text{Unit agreement} & \text{Exact process agreement} & \text{MAE / 20} \\\\ \hline \text{Expert A - Expert B} & 99.74 & 90.50 & 0.100 \\\\ \text{GPT-4.1 - Expert A} & 99.78 & 92.50 & 0.080 \\\\ \text{GPT-4.1 - Expert B} & 99.70 & 91.00 & 0.090 \\\\ \hline \end{array}$$ **Credit for alternative scientific analyses.** Sections 3.1 and 4.1 explain how report-first and time-series-first investigations are assessed through the scientific findings they establish and their supporting evidence and calculations. The rubric rewards supported findings across valid tool sequences. Appendix D.3 adds an opponent-balanced comparison of complete answers and trajectories. Its preference pattern complements the obligation-level evaluation. $$\begin{array}{lc} \hline \text{System} & \text{Win share - loss share (pp)} \\\\ \hline \text{GPT-5.6 Sol} & 67.40 \\\\ \text{GPT-4o} & -40.30 \\\\ \hline \end{array}$$ **Scoring-input review.** We also reviewed the supplementary scoring inputs to confirm that the evaluator received the corresponding investigation records and saved observations. These checks connect expert judgment, complete-output preferences, and the evidence used for scoring. The audit entries below are percentages. $$\begin{array}{lcc} \hline \text{System} & \text{Input consistency} & \text{Observation retention} \\\\ \hline \text{GPT-6.1 Terra} & 100.00 & 100.00 \\\\ \text{GPT-6.1 Sol} & 100.00 & 100.00 \\\\ \text{Gemini 3.8 Flash} & 100.00 & 100.00 \\\\ \text{Gemini 4 Argon} & 100.00 & 100.00 \\\\ \text{DeepSeek V4.1 Flash} & 100.00 & 100.00 \\\\ \text{GLM-5.3 Flash} & 100.00 & 100.00 \\\\ \text{Qwen3.8 Max} & 100.00 & 100.00 \\\\ \text{DeepSeek V4 Flash} & 100.00 & 100.00 \\\\ \text{GPT-6 Luna} & 100.00 & 100.00 \\\\ \text{GPT-5.6 Terra} & 100.00 & 100.00 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer t7Ji — Q2: Complete anonymous materials and reproducibility

Question: Can the complete evidence packages, computation scripts, and scored trajectories be inspected anonymously during review?

Thank you for requesting complete materials for independent inspection. The review supplement provides the full event evidence collection and ground-truth computation scripts, together with representative scored trajectories covering heat-wave, flood, and wildfire investigations. The evaluation repository supplies the public questions, reference answers, structured ground truth, scientific tools, solver interface, scoring prompts, and reporting code.

Anonymous review resources

The scored records include tool calls, observations, final answers, scores, and judge rationales. A reviewer can therefore follow an investigation from its public question through evidence use and calculation to the final judgment.

Section 3.2 connects scientific questions, reference solutions, structured ground truth, and executable computation. Appendix C.4 describes scoring and evaluation isolation. The accompanying materials make these connections inspectable in the implementation.

We will update the Reproducibility Statement to link directly to these review-time resources and retain the commitment to release the complete recorded evaluation trajectories.

We respectfully ask you to raise your rating in light of these clarifications and additional results. Thank you for your careful review.

Dear Reviewer t7Ji,

Thank you again for your thoughtful review and for recognizing the value of EarthVerse’s multi-source investigations and evidence-localization experiments. Your concerns focused on scoring validation and access to the review materials, which we have addressed through independent expert evaluation and anonymous access to the complete resources.

We understand these concerns to concern validation and reproducibility rather than the benchmark’s novelty or scientific design. As the discussion period is nearing its end, we respectfully ask you to consider raising your rating in light of the additional evidence.

If concerns remain that prevent a higher assessment, we would appreciate your identifying the specific methodological or benchmark-design issue so that we can address it directly. We welcome any further questions and will respond promptly.

Thank you for your time and consideration.

Reviewer_twAEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EarthVerse, a benchmark for scientific agents conducting investigations within event-specific evidence packages. It contains 405 tasks grounded in 199 events across 19 hazard families. Agents must select evidence, reconcile spatial and temporal scales, perform calculations, and explain their findings. The evaluation combines answer-unit correctness with task-specific process rubrics and compares 25 model and agent systems. Claude Fable 5 achieves the highest mean answer-unit accuracy of 84.65%, while GPT-5.6 Sol achieves the highest Strict@95 of 34.81%. Controlled experiments examine evidence access, reasoning effort, tool selection, context retention, and final review.

Critical strengths
  1. The benchmark provides multi-source analysis tasks with substantial construction and quality assurance. Reference analyses, structured answers, executable ground-truth computation programs, and task-specific scoring criteria support reproducibility and error diagnosis. Human review checks consistency among the questions, package evidence, reference answers, and scoring targets, making these resources useful for inspecting what a scientific answer should establish.

  2. The evaluation covers a broad range of systems and includes useful diagnostic experiments. The evaluated systems include GPT-5.6 Sol, Claude Fable 5, and complete agents using Codex and Claude Code. Controlled studies of evidence access, tool selection, context retention, and final review help identify specific bottlenecks in the scientific analysis workflow, making the evaluation informative beyond the leaderboard.

Critical weaknesses
  1. The tasks have limited representativeness of real scientific investigation. Several of the hardest parts of research are handled in advance: the authors find, download, and organize the data, provide some preprocessed products, and screen out known evidence defects. Some tasks also prescribe calculation metrics, thresholds, and scenario rules. Independent quality assessment, method and climatological-baseline selection, uncertainty quantification, and judging evidence sufficiency in an open setting are insufficiently evaluated. The overlap in evidence, computations, and answer units between tasks from the same event is also unquantified. Consequently, the results primarily establish analysis capabilities within curated evidence packages, providing limited support for broader scientific investigation skills.

  2. The primary scores and central reliability conclusion need stronger validation. Answer and process scores depend on a fixed GPT-4.1 judge, but the paper does not adequately justify this choice or report whether the main results remain consistent when another frontier model performs the scoring. Moreover, tasks contain about 27 answer units on average, while Strict@95 requires at least 95% of those units to be correct. Stringent aggregation across multiple units can itself substantially reduce task-level completion. The gap between mean accuracy and Strict@95 therefore does not, by itself, establish cross-step error propagation or inconsistent scientific reasoning. The effects of judge choice, task length, and aggregation thresholds need to be distinguished to substantiate the headline reliability claims.

Questions for authors
  1. Why was GPT-4.1 selected as the primary scoring model? Are results available using another frontier judge that establish the sensitivity of system rankings and main conclusions to this choice?

  2. Which aspects of data processing, quality assessment, method selection, and uncertainty analysis are actually left to the agent? What is the overlap in evidence, computations, and answer units between tasks from the same event?

  3. How does the reliability gap change when results are stratified by the number of answer units or the completion threshold is varied? Can the existing analyses distinguish aggregation effects from actual cross-step scientific errors?

14 comments
AuthorsOpenReview ↗

Response to Reviewer twAE — Q1. Judge choice and scoring reliability

Question: Why was GPT-4.1 selected, and how sensitive are scores and rankings to judge choice?

Thank you for recognizing the construction and quality-assurance work behind EarthVerse, its executable references and task-specific rubrics, and the breadth of the evaluation. We also appreciate your assessment that the diagnostic experiments provide insights beyond the leaderboard.

Section 4.1 and Appendix C.4 specify how GPT-4.1 applies the same task-specific answer contracts and process rubrics across systems. The evaluator checks scientific obligations against the references and accepts equivalent answers and valid alternative workflows.

We rescored the same saved responses with Gemini 3.1 Pro and GPT-5.6 Luna. Their answer-unit decisions closely match GPT-4.1 and preserve the ordering of the compared systems. Cohen's kappa assesses agreement beyond chance; Spearman's rho assesses ranking consistency.

Independent Earth-science experts also evaluated anonymized answers and trajectories, with system identities and automated scores hidden. Expert-judge agreement closely tracks expert-expert agreement. Appendix D.3 adds a complete-answer and trajectory comparison whose preferences broadly follow the Core ranking.

Saved-output rescoring: answer-unit accuracy (%).

\hline \text{System}&\text{GPT-4.1}&\text{Gemini 3.1 Pro}&\text{GPT-5.6 Luna}\\\\ \hline \text{Claude Fable 5}&84.65&84.66&84.63\\\\ \text{GPT-5.5}&77.21&77.23&77.19\\\\ \text{GeoMMAgent}&68.02&68.03&68.00\\\\ \text{GPT-4o}&37.99&38.00&37.98\\\\ \text{Overall}&66.97&66.98&66.95\\\\ \hline \end{array}$$ **Agreement with GPT-4.1.** $$\begin{array}{lccc} \hline \text{Judge}&\text{Unit agreement (\\%)}&\text{Cohen's }\kappa&\text{Spearman's }\rho\\\\ \hline \text{Gemini 3.1 Pro}&99.63&0.9916&1.00\\\\ \text{GPT-5.6 Luna}&99.49&0.9884&1.00\\\\ \hline \end{array}$$ **Independent expert and automated scoring agreement.** $$\begin{array}{lccc} \hline \text{Scoring pair}&\substack{\text{Unit}\\\\\text{agreement (\\%)}}&\substack{\text{Exact process}\\\\\text{agreement (\\%)}}&\text{MAE / 20}\\\\ \hline \text{Expert A - Expert B}&99.74&90.50&0.100\\\\ \text{GPT-4.1 - Expert A}&99.78&92.50&0.080\\\\ \text{GPT-4.1 - Expert B}&99.70&91.00&0.090\\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer twAE — Q2(a): Scientific work performed by the agent

Question: Which aspects of data processing and quality assessment are left to the agent?

Thank you for asking which parts of the investigation are performed by the agent. Sections 3.1–3.2 distinguish the scientific target from the candidate observations in the event package. The agent locates relevant records, aligns temporal and spatial support, performs calculations, reconciles sources, and develops the physical interpretation. Task-specific methods and scenarios are stated in the question, while the evidence-supported analysis path remains open.

Reconstructing statistics from observations. We removed derived daily products while retaining hourly observations. Both models’ tool traces show that they read the hourly records and recomputed the required statistics. The table reports accuracy on the resulting scientific targets (%).

\hline \text{Model} & \text{Task} & \text{Original} & \text{Products removed} & \Delta\text{ (pp)} \\\\ \hline \text{DeepSeek V4.1 Flash} & \text{CSX-013 Q2} & 100.00 & 83.33 & -16.67 \\\\ \text{DeepSeek V4.1 Flash} & \text{CSX-025 Q1} & 100.00 & 100.00 & 0.00 \\\\ \text{GLM-5.3 Flash} & \text{CSX-013 Q2} & 83.33 & 66.67 & -16.67 \\\\ \text{GLM-5.3 Flash} & \text{CSX-025 Q1} & 100.00 & 100.00 & 0.00 \\\\ \hline \end{array}$$ **Assessing source quality.** We compared the original package with an added innocuous candidate and an added candidate containing a time-window, spatial-scope, or unit-label defect. The valid original source remains available. Source exposure and explicit detection are recorded separately. The experiments use fixed targets and a common diagnostic grader, with solver identities and condition labels hidden. $$\begin{array}{lcccccc} \hline \text{Model} & \text{Original accuracy} & \text{Innocuous accuracy} & \text{Defective accuracy} & \text{Defective source read} & \text{Explicit detection} & \text{Detection given read} \\\\ \hline \text{DeepSeek V4.1 Flash} & 73.22 & 73.26 & 73.82 & 29.17 & 0.00 & 0.00 \\\\ \text{GLM-5.3 Flash} & 69.72 & 65.46 & 74.42 & 8.33 & 4.17 & 50.00 \\\\ \hline \end{array}$$ All entries in the source-quality table are percentages. These interventions connect the final scientific answer with the reconstruction and evidence-checking decisions made during the investigation.
AuthorsOpenReview ↗

Response to Reviewer twAE — Q2(b): Evidence sufficiency and task relationships

Question: How is evidence sufficiency evaluated, and how are related tasks reviewed?

Thank you for raising evidence sufficiency and reuse across tasks. We removed critical sources and known duplicates, with irrelevant-file deletion as the control. The evaluation credits supported qualifications, unavailable results, and conclusions grounded in the retained evidence. Evidence-insufficiency recognition and unsupported assertions are recorded separately and can occur in the same answer. This tests whether an agent carries its evidence assessment into its conclusion.

Evidence-sufficiency intervention (%).

\hline \text{Model} & \text{Condition} & \text{Insufficiency recognized} & \text{Unsupported claims} \\\\ \hline \text{DeepSeek V4.1 Flash} & \text{Original} & 5.00 & 5.00 \\\\ \text{DeepSeek V4.1 Flash} & \text{Irrelevant removed} & 10.00 & 5.00 \\\\ \text{DeepSeek V4.1 Flash} & \text{Critical removed} & 70.00 & 85.00 \\\\ \text{GLM-5.3 Flash} & \text{Original} & 5.00 & 10.00 \\\\ \text{GLM-5.3 Flash} & \text{Irrelevant removed} & 10.00 & 25.00 \\\\ \text{GLM-5.3 Flash} & \text{Critical removed} & 65.00 & 85.00 \\\\ \hline \end{array}$$ **Review of related tasks.** We reviewed the source files and scientific requirements of related tasks. Each event package supports its own investigation, and tasks sharing an event background address distinct scientific targets, with modest reuse of answers across tasks. **Core under alternative aggregation.** $$\begin{array}{lccc} \hline \text{Model} & \text{Task-equal Core} & \text{Key-equal Core} & \Delta\text{Core} \\\\ \hline \text{GPT-6.1 Sol} & 87.25 & 86.73 & -0.52 \\\\ \text{GPT-5.6 Terra} & 82.78 & 82.69 & -0.10 \\\\ \text{GPT-6.1 Terra} & 79.34 & 80.09 & 0.76 \\\\ \text{Gemini 4 Argon} & 75.93 & 76.51 & 0.58 \\\\ \text{DeepSeek V4.1 Flash} & 72.25 & 72.57 & 0.32 \\\\ \text{GPT-6 Luna} & 64.99 & 64.23 & -0.76 \\\\ \text{Gemini 3.8 Flash} & 63.65 & 63.77 & 0.12 \\\\ \text{DeepSeek V4 Flash} & 60.06 & 58.49 & -1.58 \\\\ \hline \end{array}$$ The Core ordering remains unchanged when results are aggregated equally by task or by existing canonical event groups. This further supports the consistency of the model comparison.

Comments on the paper

1 comment
AuthorsOpenReview ↗

EarthVerse: Final Rebuttal Summary and Request for Independent Assessment

Dear Area Chair and Reviewers,

Thank you for reading EarthVerse and engaging with the rebuttal. Reviewer twAE recognized the care taken in constructing and checking the benchmark, the executable reference analyses, and the diagnostic value beyond the leaderboard. Reviewer t7Ji highlighted the multi-source investigations and paired evidence-localization experiments. Reviewer Uo12 noted the benchmark’s practical relevance, evaluation scale, and range of experimental settings.

EarthVerse evaluates the scientific work between finding an observation and reaching a conclusion. Agents start with a heterogeneous event archive and must decide which records apply, reconcile differences in time and spatial coverage, perform the necessary calculations, and explain what the evidence supports. This brings the evaluation closer to real hazard analysis, where several records may be relevant to an event but only some can support a particular inference. Executable reference calculations and task-specific rubrics let us check that work while allowing different valid approaches.

The Kīlauea example shows what this design reveals. Two tasks share the same archive and support the same diagnosis, but one uses lava effusion rate to compare mechanisms, while the other requires the ratio of summit-collapse volume to lava-flow volume. The model identified the right mechanism and reported the wrong quantity. A score based only on the final diagnosis would miss that error. The difference is also visible across the release: the 405 tasks span 27 capability combinations, and 72% of pairs within the same hazard family but with different event keys emphasize different primary capabilities.

What the Rebuttal Added

  • Scoring reliability and access. We added blinded expert assessment and rescoring with alternative judges, clarified access to the full benchmark materials, and pointed to the construction details and source inventory already in the main text and appendices.
  • Data processing and evidence quality. We observed agents reconstructing daily statistics from hourly records after removing the derived summaries. We also tested how agents handle defective candidate sources and whether they recognize when key evidence is missing.
  • Task diversity and scientific dependencies. We examined coverage across the full release and compared the reference procedures for tasks that share an archive. We varied completion thresholds and examined task length and aggregation. We also corrected specific upstream errors to see which downstream findings changed.
  • Newer models on matched tasks. We evaluated models released after the submission deadline on matched subsets within the review period’s time and computational budget. Strict@95 improved in the supplementary comparison, yet even GPT-6.1 Sol achieved near-complete success on only about three-fifths of the evaluated tasks. This reinforces a central finding of the original full-benchmark evaluation: strong average performance on individual requirements can coexist with substantial difficulty completing an investigation.

We responded to each concern with a clarification, a pointer to material already in the submission, or a targeted analysis. These additions reinforce the paper’s findings and extend its diagnostic experiments. The benchmark methodology and original main experiments are unchanged. The new evidence helps explain why models succeed or fail under the design already presented in the paper.

Request to the Area Chair

We appreciate that twAE explicitly identified the scoring-reliability and Q3 concerns as resolved. We are concerned that other follow-ups acknowledged that some or most issues had been addressed, yet did not explain which specific concern still determined the recommendation or how the new evidence was weighed. After a point-by-point rebuttal and substantial additional analysis, those brief responses leave us unsure whether the paper’s main contribution and the evidence supporting it received full consideration.

We ask the Area Chair to assess the submission and rebuttal independently. If a concern still affects the decision, we would appreciate knowing which claim it bears on and why the available evidence does not address it. We respect that the final judgment may differ from ours; we ask that it reflect the work in the submission and what the discussion established.

Thank you for your time and consideration.

Best regards,
The EarthVerse authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.