acceptodds
Withdrawn from ICLR 2027

EarthArena: Benchmarking Agents on Real-world, Long-Horizon Scientific Tasks in Earth Observation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Scientific investigations in Earth observation combine heterogeneous observations, geospatial processing, and interdisciplinary analyses into long-horizon workflows with interdependent stages. Agents must apply scientific methods appropriately and maintain the scientific validity of intermediate artifacts and final conclusions. However, whether they can meet these requirements throughout the workflow remains to be systematically evaluated. To this end, we introduce EarthArena, an executable benchmark for real-world, long-horizon scientific tasks in Earth observation. We construct 101 tasks from 30 peer-reviewed publications spanning diverse areas of Earth observation. Each case organizes tasks linked by intermediate artifacts around a scientific objective, with reproduced reference results and human-reviewed scoring rubrics. EarthArena links scientific conclusions to supporting artifacts and execution evidence to assess agents' ability to maintain scientific validity throughout the workflow. Across six language models and three agent harnesses, the best-performing configuration achieves a case-averaged Rubric score of 44.3. Model rankings vary across harnesses, highlighting the importance of evaluating scientific task performance at the level of complete model-harness configurations. EarthArena provides a testbed for systematically evaluating and improving agents' ability to conduct real-world, long-horizon scientific investigations in Earth observation.

Primary area:
datasets and benchmarks
Keywords:
Earth observation, language agents, agent evaluation, datasets and benchmarks, long-horizon tasks, scientific workflows

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tsNvOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EARTHARENA, an executable benchmark for testing whether agents can carry out long, multi-step investigations in Earth observation. It includes 30 cases and 101 linked tasks based on peer-reviewed papers. Each case is organized as a dependency graph: earlier data products and analyses provide the basis for later scientific conclusions. The evaluation covers six models and three harnesses. The strongest reported configuration, GPT-6-Luna with the DeepSeek harness, achieves a case-averaged Rubric score of 44.3.

Critical strengths
  • The paper raises a question that existing benchmarks do not examine very well: can an agent maintain scientific validity throughout a full Earth observation workflow, rather than simply answer a series of isolated questions? The task dependencies reflect how errors can carry forward in real work.

  • The benchmark, which looks at answers, artifacts, and the process used to produce them, goes beyond existing evaluations that score only the final answer by also assessing the artifacts and process used to produce it.

  • The authors provide a balanced discussion of the benchmark’s limitations. They discuss the difficulty of open-world data discovery, the limited coverage of inaccessible or proprietary inputs, the challenge of separating model effects from infrastructure effects, and the need for more extensive auditing of the judge.

Critical weaknesses
  • Some important settings are not explained well enough. The evaluator assigns fixed weights of 0.5, 0.3, and 0.2 to Answer, Artifact, and Process, and uses a 60% frozen leaf identifier matching threshold. The paper does not give a principled reason or outside reference. Since these choices directly affect partial credit and could change the model rankings, the authors should explain them.
  • Lacking of randomness and reproducibility. The manuscript does not report LLM settings such as temperature. It also gives no results from independent repeated runs or confidence intervals for the 18 configurations. Retries within a single run do not provide the same information as independent trials. This is especially relevant when the differences are small: the two best configurations under the DeepSeek harness are separated by only 2.3 Rubric points.
  • I suggest that the authors should release a task-level scoring manifest with branch applicability, fallback indicators, task weights, branch scores, and the relevant leaf-level aggregation details.
  • The 30 cases are distributed unevenly across domains. Atmosphere contributes 14 cases (46.7%), while Land Surface contributes only one, and several other domains contribute just two or three cases each. The authors acknowledge that the domain breakdown is still incomplete, but they do not report performance by domain.
Questions for authors

See weakness.

Reviewer_j1nnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces EarthArena, an executable benchmark designed to evaluate LLM agents on real world scenery, long-horizon scientific investigations in Earth observation. By constructing 101 tasks from 30 publications across diverse geoscience domains, the authors establish a directed acyclic dependency graph where upstream artifacts constrain downstream analyses, thereby requiring agents to maintain scientific validity throughout complex workflows. The evaluation of six models across three agent harnesses reveals that performance is highly configuration dependent, with the top performing system achieving a Rubric score of 44.3, while failure mode analysis identifies sustained delivery of scientific outputs and scale aware data acquisition as primary challenges for autonomous research agents.

Critical strengths

EarthArena establishes a paper-grounded testbed for evaluating agents on long-horizon Earth observation investigations. By constructing 101 tasks from 30 peer-reviewed publications, the authors organize scientific objectives into directed acyclic dependency graphs where upstream artifacts constrain downstream analyses. This realistic structure, coupled with human-reviewed scoring rubrics that link conclusions to execution evidence, reveals that maintaining scientific validity throughout complex workflows depends heavily on the integrated model–harness configuration.

Critical weaknesses
  1. Insufficient validation of harness induced variability across diverse scientific domains: The authors find that model rankings fluctuate significantly across the three evaluated harnesses, but the analysis fails to isolate which specific harness mechanisms account for these performance shifts. While Table 2 and Figure 5 identify aggregate failure modes, the experimental design does not provide controlled ablations to determine if a harness improves scientific validity through better data partition planning or more robust dependency recovery. Consequently, it remains unclear whether the reported Rubric scores reflect an agent's inherent scientific reasoning or are merely artifacts of how a specific harness handles the large-scale data requirements mentioned in Section 6.1.

  2. The construction of EarthArena relies on LLM-assisted screening and agent-assisted reference execution to define gold results. This creates a potential bias where the benchmark might only include tasks that existing models find representable or executable. By filtering 10,636 publications down to 30 cases based on "recoverable data and methods" via LLMs, the authors may have inadvertently excluded the most complex scientific workflows that current AI cannot yet formalize.

  3. The paper evaluates model harness configurations against reference artifacts produced by similar agent-driven processes. This setup risks a logical circularity where the "gold standard" is defined by the same architectural paradigms being tested. If the reference-generating harness shares systematic flaws with the test harnesses, such as specific geospatial processing biases, the Rubric score might reward conformity to an agentic trace rather than objective scientific validity.

Questions for authors
  1. Provide a controlled ablation study on a subset of cases to isolate the marginal contribution of specific harness features, such as resumable downloads or explicit parameter persistence, to the overall Rubric score?

  2. What measures were taken to ensure that the reference execution remains an objective scientific baseline rather than a reflection of the specific tool-use patterns and limitations inherent in the agent harnesses used to generate it?

  3. How did the authors verify that the LLM-assisted screening did not systematically exclude high-complexity scientific dependencies that are present in the original 10,636 publications but currently beyond the representational limits of LLMs?

Reviewer_oYVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EarthArena, an executable benchmark consisting of 101 tasks organized into 30 paper-grounded Earth observation cases. Tasks are connected through intermediate artifacts, and evaluation considers answers, artifact correctness, and execution evidence. Experiments evaluate six language models across three agent harnesses, reporting a best case-averaged Rubric score of 44.3 and substantial performance differences across model–harness configurations.

Critical strengths
  1. The benchmark provides executable scientific tasks grounded in real publications. Its 101 tasks come from 30 Earth observation papers and include reproduced reference results and human-reviewed scoring rubrics. This scientific grounding makes the benchmark a valuable resource for evaluating agents on analyses involving real, heterogeneous observations.

  2. The artifact-linked task structure and evidence-based evaluation are meaningful design choices. Connecting tasks through intermediate products and considering answers, artifacts, and execution records provides a way to check whether conclusions are supported by actual computations. This offers a useful structure for assessing scientific consistency across dependent analysis steps.

Critical weaknesses
  1. The evaluation protocol and reliability of the reported scores are insufficiently validated. Table 1 highlights multi-turn interaction and user simulation, but the main text does not adequately specify feedback generation, the simulator's information access, or its separation from private references. The human scoring audit covers only answer branches in 10 tasks, leaving artifact, process, and complete-case scoring insufficiently validated. Moreover, Qwen3.6-35B-A3B scores zero across all performance metrics under Codex but obtains nonzero scores under the other harnesses, without an explanation distinguishing task failures from compatibility, submission, or evaluation problems. Together, these gaps limit confidence in the interpretation of the main scores and system comparisons.

  2. The experiments provide insufficient direct validation of long-horizon scientific workflows. The paper emphasizes dependencies between intermediate artifacts and scientific validity throughout an investigation, but mainly reports aggregate scores. It does not report complete-case success rates or analyze how upstream errors propagate, are detected, or are corrected. Consequently, the experiments do not sufficiently establish whether agents maintain scientific validity across dependent analysis steps, limiting support for the benchmark's central long-horizon evaluation claim.

Questions for authors
  1. What information can the user simulator access? Does feedback returned to the agent contain information derived from private reference answers or hidden scoring criteria?

  2. Beyond the reported answer-branch audit, is there candidate-level human validation of artifact, process, and complete-case scoring? If so, how closely do these scores agree with human judgments?

  3. What is the end-to-end case success rate in the existing runs, and what criterion defines success? How do upstream artifact errors affect downstream task inputs and scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.