- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents EARTHARENA, an executable benchmark for testing whether agents can carry out long, multi-step investigations in Earth observation. It includes 30 cases and 101 linked tasks based on peer-reviewed papers. Each case is organized as a dependency graph: earlier data products and analyses provide the basis for later scientific conclusions. The evaluation covers six models and three harnesses. The strongest reported configuration, GPT-6-Luna with the DeepSeek harness, achieves a case-averaged Rubric score of 44.3.
Critical strengths
-
The paper raises a question that existing benchmarks do not examine very well: can an agent maintain scientific validity throughout a full Earth observation workflow, rather than simply answer a series of isolated questions? The task dependencies reflect how errors can carry forward in real work.
-
The benchmark, which looks at answers, artifacts, and the process used to produce them, goes beyond existing evaluations that score only the final answer by also assessing the artifacts and process used to produce it.
-
The authors provide a balanced discussion of the benchmark’s limitations. They discuss the difficulty of open-world data discovery, the limited coverage of inaccessible or proprietary inputs, the challenge of separating model effects from infrastructure effects, and the need for more extensive auditing of the judge.
Critical weaknesses
- Some important settings are not explained well enough. The evaluator assigns fixed weights of 0.5, 0.3, and 0.2 to Answer, Artifact, and Process, and uses a 60% frozen leaf identifier matching threshold. The paper does not give a principled reason or outside reference. Since these choices directly affect partial credit and could change the model rankings, the authors should explain them.
- Lacking of randomness and reproducibility. The manuscript does not report LLM settings such as temperature. It also gives no results from independent repeated runs or confidence intervals for the 18 configurations. Retries within a single run do not provide the same information as independent trials. This is especially relevant when the differences are small: the two best configurations under the DeepSeek harness are separated by only 2.3 Rubric points.
- I suggest that the authors should release a task-level scoring manifest with branch applicability, fallback indicators, task weights, branch scores, and the relevant leaf-level aggregation details.
- The 30 cases are distributed unevenly across domains. Atmosphere contributes 14 cases (46.7%), while Land Surface contributes only one, and several other domains contribute just two or three cases each. The authors acknowledge that the domain breakdown is still incomplete, but they do not report performance by domain.
Questions for authors
See weakness.