acceptodds
Under review as a conference paper at ICLR 2027

EarthDataBench: Benchmarking LLM Agents for Long-Horizon Scientific Data Production

Abstract

Transforming heterogeneous Earth data into reliable scientific products requires complex, labor-intensive processing, limiting the pace and scale of research. Although large language model (LLM) agents have demonstrated success in automating a range of scientific tasks, their ability to produce research-quality Earth-science data remains insufficiently assessed. We introduce EarthDataBench, a benchmark of 40 executable, paper-derived tasks spanning seven Earth-science topics and diverse data sources, processing mechanisms, and product types. Tasks require agents to complete long-horizon scientific workflows, integrating heterogeneous data and implementing domain-specific methods to produce research-quality datasets. Unlike existing QA benchmarks that rely on matching reference answers, our evaluation combines comparisons with expert products and validation against independent observations, enabling assessment of whether agent-generated products match or surpass expert-reference quality. Across nine LLMs and ten model–harness configurations, 52.0% of outcomes pass scientific-consistency checks, but only 19.3% reach expert-reference quality or higher and 0.8% outperform expert references on independent validation. Our empirical results suggest that reliable execution remains a key bottleneck in scientific data production, even when methods are prescribed. EarthDataBench provides a testbed for advancing research-ready data production in Earth science and assessing improvements beyond expert references.

Primary area:
datasets and benchmarks
Keywords:
Earth science data, LLM Agent, Benchmark, Dataset production

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XNjpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EarthDataBench: dozens of long-horizon tasks, curated from papers in an Earth-science data journal, that require turning raw observational data into research-quality data products across several Earth-science domains. Evaluation uses a five-level cumulative score, from "usable deliverables" through "scientifically consistent" and "at expert level" to "surpassing the expert product on independent observations." Evaluating several LLMs under two agent frameworks, the authors find that about half of the outcomes are scientifically consistent, about one fifth reach expert level, almost none surpass the expert, and format problems are the leading source of failure.

Critical strengths
  1. Valuable topic, clearly distinct from existing Earth-science benchmarks. Requiring agents to deliver complete data products (with code and reports) from raw data is much closer to real scientific work than the common QA-style evaluations.
  2. Expert involvement and calibration awareness in construction. Reference products are manually verified, and the task cards are thoroughly disclosed, making it possible to audit the evaluators themselves.
Critical weaknesses
  1. The unified scoring protocol promised in the main text is not implemented uniformly across the tasks' actual rules, so the headline numbers are not on a common footing. The main text says the five levels are strictly cumulative and that "expert level" requires both closeness to the expert product and being no worse than the expert on independent observations. But in the appendix task cards: some scores are not cumulative (in the Biosphere task, the 0.8 level requires similarity S ≥ 0.90, lower than the 0.6 level's S ≥ 0.92, so a product can earn 0.8 without qualifying for 0.6); only about three tasks actually check both conditions; the "expert reference" is sometimes the published product, sometimes an AI-assisted reconstruction, and sometimes just a baseline model; and four tasks list thresholds as "not provided." As a result, the abstract's "52% pass scientific checks, 19.3% reach expert level" does not mean the same thing across tasks and needs to be recomputed under a unified standard.

  2. The "beyond expert" level is barely measurable in the current data, and single-run results plus unspecified run conditions add uncertainty. Only 3 of 400 outcomes reach "beyond expert," and the top level is 0; some tasks are hard to surpass by design (e.g., tasks that require copying native values directly). All results come from a single run (each task is worth 2.5 percentage points, so the 7.5-point framework difference amounts to 3 tasks, and inter-model differences are of the same magnitude); the paper does not state whether the execution environment has network access (the tasks come from an open-access journal, so agents could potentially fetch the published products directly), nor the hardware or time budgets.

Questions for authors

Clarifications needed

  1. Please provide the level-by-level decision rules for all 40 tasks: for how many tasks does "expert level" check both consistency with the expert product and independent observations? For how many is the scoring not strictly cumulative? If recomputed under a unified standard, how would the abstract's 52.0% / 19.3% change?
  2. Which tasks do the 3 "beyond expert" results come from? For each task, have you verified that "beyond expert" is achievable in principle (e.g., with an oracle or a known better method)?
  3. Does the execution environment have network access? Have you checked whether agents directly download the published products?

Suggestions

  • Add comparisons with general science / data-science agent benchmarks such as ScienceAgentBench, MLE-bench, and DSBench.
  • Report format-related zero scores separately to distinguish "delivery problems" from "scientific capability."
  • Provide at least a few complete task packages (with evaluators and thresholds) before release to support community reproduction.
4 comments
AuthorsOpenReview ↗

Scoring and Reproducibility (1/2)

We thank the reviewer for recognizing the value of artifact-level evaluation and expert involvement in EarthDataBench.

We have revised the scoring rules and reference documentation, updated the results, and expanded the Base evaluation to 1,000 outcomes. We also added repeated evaluations, separated delivery failures from scientific failures, and documented the offline sandbox and per-task resource limits.

W1 / Q1 — Consistency of the scoring protocol and revised results

Concern summary: Scoring consistency, reference provenance, threshold disclosure, and revised results.

We completed a systematic review and update of the scoring rules, evaluators, and task briefs for all 40 tasks.

(a) Cumulative scoring

All 40/40 tasks now use strictly cumulative scoring, and 40/40 require both agreement with the expert product C and non-inferiority on independent observations B at 0.6. Higher levels inherit all preceding requirements under the common framework (Section 3.3, pp. 6–7). The Biosphere inconsistency is corrected: 0.8 now retains the 0.6 requirements of S ≥ 0.92 and F ≥ 0.95 (Appendix A.16, p. 31). The revised task-specific rules are provided for all 40 tasks (Appendix A, pp. 15–55).

(b) Reference products and thresholds

For 39 tasks, references are author-released products or their deterministic derivatives. ChinaAI-FSC uses an explicitly labelled AI-assisted reconstruction of the author's UNet method because the original weights and test predictions are unavailable (Appendix A.10, p. 25). The task briefs clarify these sources and replace the former “not provided” entries with task-specific scoring rules and threshold descriptions (Appendix A, pp. 16–55). Remaining private numerical thresholds are explicitly identified.

(c) Revised results

The revised evaluation preserves the main finding that scientific consistency is substantially more common than expert-reference quality. The original submission and current expanded evaluation are summarized below.

EvaluationOutcomesScientific consistency ≥0.4Expert-reference quality ≥0.6Beyond-reference quality ≥0.8
Original submission40052.0%19.25%0.75%
Full Base1,00059.2%15.0%0.7%

The full Base row reflects both revised scoring and additional experiments. It covers 1,000 outcomes across 25 result sets, 17 model–harness configurations, and 11 LLMs (Section 4.2, p. 8 and Table 3, p. 9). These results retain the central conclusion: reliable execution remains a bottleneck even when scientific methods are prescribed.

S1 — Comparison with general scientific and data-science benchmarks

Concern summary: Comparison with ScienceAgentBench, MLE-bench, and DSBench.

We have added discussion and comparisons of general scientific and data-science benchmarks, including ScienceAgentBench, MLE-bench, and DSBench, in the main text (Section 2, pp. 2–3 and Table 1, p. 3).

S2 — Separating delivery failures from scientific-quality failures

Concern summary: Separation of delivery errors from scientific-quality failures.

Delivery failures and scientific invalidity are now diagnosed separately. Of the 181 zero-score outcomes, 140 fail before scientific scoring and 41 fail minimum scientific validity after delivery (Section 4.3, p. 10, and Appendix C.1, Table 9, p. 59). The updated appendix identifies the delivery failure or first unmet scientific requirement for all 1,000 Base outcomes, including barriers at higher score levels (Appendix C.1 and Table 10, p. 59). The LST example illustrates successful delivery with a zero scientific score because predictions extend outside the permitted spatial support (Appendix D.5, p. 65).

S3 — Reproducible task packages

Concern summary: Access to complete task packages, evaluators, and thresholds for reproduction.

We have provided one complete representative task package from each Earth-science domain, including evaluators and scoring thresholds. The packages are available at the anonymous sample repository.

AuthorsOpenReview ↗

Results and Reliability (2/2)

W2 / Q2 — Beyond-reference outcomes

Concern summary: Beyond-reference outcomes and the attainability of higher scores.

We address both score-level attainability and the observed beyond-reference outcomes. The outcome analysis has been added to the revised manuscript (Appendix C.3, p. 60).

Beyond-reference attainability was verified during task construction. We used expert-reference products and constructed oracle test cases to calibrate each task evaluator across score levels. The oracle tests confirmed that the cumulative requirements for 0.8 and 1.0 could be satisfied (Section 3.4, p. 7).

(a) Original beyond-reference outcomes

The original three beyond-reference outcomes came from two tasks. Two were HUST-GRACE reconstructions by GPT-6-Astra and GPT-6-Luna with Codex, and one was TWSTORE with GLM-5.3-Flash/Claude Code. Under the revised rules, both HUST-GRACE outcomes score 0.6 because their C20 improvements fail the 12-month moving-block bootstrap test required for 0.8 (Appendix A.39, p. 54).

The TWSTORE outcome retains 0.8, but does not reach 1.0 because the required improvement in the 2011–2016 period is not achieved. The temporal validation requirements are specified in Appendix A.29 (p. 44).

(b) Additional evidence from new experiments

The expanded Base cohort contains seven 0.8 outcomes: six on TWSTORE and one on ID72, with none at 1.0 (Appendix C.3, p. 60). For TWSTORE, the GLM-5.3/Claude Code example reported in the manuscript reduces independent 1993–2016 centered RMSE from 9.82 to 6.85 mm. All six TWSTORE outcomes fail the stronger improvement requirements in at least one early or late validation period.

For ID72 forest-height mapping, GPT-6-Astra/Codex combined terrain-corrected radar features with GEDI height samples and selected a gradient-boosted regressor using spatially blocked validation. Independent airborne-lidar mean absolute error decreases by 12.2% at Howland and 14.7% at Harvard relative to the expert reference. Tall-canopy errors fail the non-inferiority requirement for 1.0, so the result remains at 0.8 (Appendix C.3, “Forest-height mapping,” p. 60).

Most expert-level outcomes still lack sufficient evidence of improvement beyond the reference. Among the 143 outcomes that stop at 0.6, 122 fail the independent-improvement requirement and 21 fail robustness or statistical-evidence requirements (Appendix C.1 and Table 10, p. 59). Improvement headroom remains task-dependent. In the exact-assembly example, reproducing the expert product yields 0.6 but no independent-validation gain (Appendix D.7, p. 67).

W2 — Reliability of repeated evaluations and model comparisons

Concern summary: Reliability of single-run results and small differences between configurations.

To strengthen reliability, we evaluated four representative model–harness configurations three times each on the same 40 Base tasks: GPT-6-Astra, GLM-5.3, and Kimi K3.1 Pro with Codex, and GLM-5.3 with Claude Code. Time and computational costs prevented us from repeating every configuration. We now report individual results, means, and sample standard deviations (Appendix B.3, pp. 57–58, Table 7, p. 58). Mean-score standard deviations range from 0.29 to 1.15 points on the 0–100 scale, indicating stable aggregate performance for these configurations. Closely performing configurations can nevertheless change order between runs, so small differences do not establish definitive rankings.

Across all 12 evaluations, scientific-consistency attainment exceeds expert-reference quality by 42.5–57.5 percentage points. This persistent gap supports our central claim that reliable scientific execution remains a key bottleneck even when methods are prescribed (Section 4.3, p. 10).

W2 / Q3 — Execution conditions and potential product downloading

Concern summary: Execution settings and the risk of retrieving published products.

We have added the execution environment and resource limits for all 40 tasks (Appendix E, Table 12, p. 68). Agents run in a sandbox without network access, so they cannot retrieve or download published products online. Public inputs and writable outputs are accessible, while independent observations, expert references, and evaluators remain outside the sandbox. CPU, memory, and time limits are configured separately for each task.

AuthorsOpenReview ↗

Revised Manuscript and Key Updates

We have revised the main text and appendices and submitted an updated manuscript. The main revisions include:

  • Expanded evaluation to 1,000 Base outcomes, including repeated runs.
  • Standardized scoring rules and clarified reference sources and thresholds.
  • Added benchmark comparisons, failure analyses, and reproducibility details.

Regarding your specific concerns:

  • All 40 tasks now follow the cumulative scoring rule, and the Biosphere inconsistency has been corrected.
  • Repeated evaluations and revised beyond-reference analyses clarify result stability and barriers to higher scores.
  • Delivery and scientific failures are distinguished, with offline execution conditions documented and representative task packages provided.

We would appreciate your review of the revised manuscript and our responses.

AuthorsOpenReview ↗

Request for Feedback and Reconsideration

Dear Reviewer XNjp,

Thank you again for your time and thoughtful feedback. As the rebuttal period draws to a close, we would appreciate your review of our revised manuscript and responses. We briefly summarize the updates below:

  • W1/Q1 (Scoring): Unified cumulative scoring across all 40 tasks, clarified references and thresholds, and updated results to 1,000 outcomes.
  • W2/Q2 (Higher scores): Clarified evaluator calibration and analyzed beyond-reference outcomes and barriers to higher scores.
  • W2 (Reliability): Added three runs for four configurations, supporting the main conclusion while qualifying close rankings.
  • W2/Q3 (Execution): Documented offline execution, evaluation-data isolation, and per-task resource limits.
  • S1 (Comparisons): Added comparisons with ScienceAgentBench, MLE-bench, and DSBench.
  • S2 (Failures): Separated 140 delivery failures from 41 scientific failures.
  • S3 (Reproducibility): Provided seven representative task packages, including evaluators and thresholds.

We would be grateful to hear whether any concerns remain. If these revisions address your concerns, we would appreciate it if you could reconsider your assessment and score. Thank you again for your time and consideration.

Reviewer_J5PjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EarthDataBench, a benchmark of 40 paper-derived, long-horizon tasks across seven Earth-science topics. The tasks are designed to evaluate the ability of LLM agents to transform heterogeneous source inputs into research-grade scientific datasets. The authors developed a six-stage human–AI workflow and a five-level cumulative scoring protocol that combines expert-product comparisons with withheld independent observational validation. Additionally, they evaluate nine LLMs across ten configurations using Codex and Claude Code. The main finding is that only 19.3% of outcomes reach expert-reference quality and 0.8% surpass it, demonstrating that fine-grained numerical execution and format compliance remain primary bottlenecks even when scientific methods are prescribed.

Critical strengths
  1. The benchmark moves beyond simple question answering to test whether agents can execute full, multi-step workflows from published Earth-science papers and produce reusable datasets.
  2. The evaluation design combines comparisons against expert datasets with validation against independent real-world observations, making it possible to test whether agents match or improve upon human expert products.
  3. Authors offer careful diagnostic, comparing guided and unguided settings
  4. Clear Visual Presentation
Critical weaknesses
  1. Results rely on a single run per task, test the second agent harness on only one model, omit domain-specialized and open-source scientific agent baselines, and do not detail sandbox limits (e.g. internet blocking /timeouts)
  2. I suspect potential memorization of the source papers. This should be discussed in the limitations sections
  3. There is no breakdown of why most passing runs stall just below expert quality, no analysis of the three runs that actually beat the expert baseline
  4. The main text describes a strict bottom-up scoring ladder, but several tasks in the appendix use top-down branching (the strict bottom-up cumulative scoring ladder is defined in the Section 3.3 and Appendix A deviate from this)
Questions for authors
  1. How much the scores improve if dataset accuracy is scored separately from markdown report and figure formatting requirements?
  2. How stable are the rankings across multiple runs per task?
  3. How do open-source or domain-specialized scientific agents perform on the benchmark?
  4. How many tasks require fusing multiple raw inputs versus using pre-processed grids with upstream sensor labels and how does a simple grid-copying task fit the "days to weeks" effort claim?
7 comments
AuthorsOpenReview ↗

Repeats and Baselines (1/4)

We thank the reviewer for recognizing the value of executable scientific workflows, independent observational validation, and the Base/Open comparison.

We have expanded the Base evaluation to 1,000 recorded outcomes across 25 result sets, 17 model–harness configurations, and 11 LLMs, including repeated evaluations and additional scientific-agent baselines. We have also standardized the cumulative scoring rules, added analyses of failures and beyond-reference outcomes, and documented the execution environment and per-task resource limits.

W1 / Q2 — Repeated evaluations and ranking stability

Reviewer concern: How stable are the rankings across multiple runs per task?

We agree that a single run is insufficient to establish reliable model rankings. Within the time and computational budget available for revision, we expanded the evaluation to three runs for each of four representative configurations on the same 40 Base tasks: GPT-6-Astra, GLM-5.3, and Kimi K3.1 Pro with Codex, and GLM-5.3 with Claude Code.

ConfigurationBase tasksEvaluationsMean Score ± SD
GPT-6-Astra / Codex40335.50 ± 1.00
GLM-5.3 / Codex40334.17 ± 0.29
Kimi K3.1 Pro / Codex40335.50 ± 0.87
GLM-5.3 / Claude Code40332.83 ± 1.15

The repeated evaluations show modest variation in aggregate performance. Mean-score standard deviations range from 0.29 to 1.15 points on the 0–100 scale. Individual runs can nevertheless change the relative ordering of closely performing configurations, so small differences should not be interpreted as definitive rankings.

These ranking fluctuations do not change our main finding that reliable scientific execution remains a bottleneck, even when methods are prescribed. Across all 12 evaluations, scientific-consistency attainment exceeds expert-reference attainment by 42.5–57.5 percentage points. This persistent gap shows that many outputs pass basic scientific checks but still fail to meet expert-reference quality requirements. The central conclusion therefore holds across the evaluated configurations and runs, rather than depending on which model ranks first.

These results are discussed in Section 4.3 (p. 10). Individual evaluation results, means, and standard deviations are provided in Appendix B.3 (pp. 57–58) and Table 7 (p. 58).

W1 / Q3 — Additional harnesses and scientific-agent baselines

Reviewer concern: The second harness was tested on only one model, and open-source or domain-specialized scientific agents were missing.

We have expanded the comparison with general scientific harnesses and a domain-adapted agent baseline. We added Open-Science (Synthetic Sciences, 2026) and EvoScientist (Zhang, 2026), both using GLM-5.3. We also evaluated a GLM-5.3/Codex agent augmented with three Earth-science skills from Scientific Agent Skills (K-Dense Inc., 2026): GeoMaster, GeoPandas, and Marine Carbonate Chemistry. We expanded the Claude Code evaluation from one to four models: GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash, and Kimi K3.1 Pro. The table below compares the five configurations using GLM-5.3 on the same 40 Base tasks.

Baseline typeConfigurationScoreScientific consistency ≥0.4Expert-reference quality ≥0.6
General-purpose coding harnessCodex34.261.7%19.2%
General-purpose coding harnessClaude Code32.865.0%15.8%
General scientific harnessOpen-Science15.032.5%2.5%
General scientific harnessEvoScientist29.555.0%15.0%
Domain-adapted agentCodex + Earth-science skills34.065.0%12.5%

Codex and Claude Code report means over three evaluations, while the other configurations each have one evaluation.

The scientific harnesses do not show an overall advantage in this comparison. With GLM-5.3, Open-Science and EvoScientist achieve lower mean scores and expert-reference attainment than Codex. Harness effects also vary by model in the Codex–Claude Code comparison.

Earth-science skills yield mixed gains. Scientific-consistency attainment rises from 61.7% to 65.0%, with one beyond-reference outcome. However, mean Score remains nearly unchanged, and expert-reference attainment falls from 19.2% to 12.5%. These results suggest that domain guidance alone does not ensure expert-quality products.

The expanded baselines and harness comparisons are described in Sections 4.1–4.2 (pp. 8–9), with results in Table 3 (p. 9). The table also reports 130.86M recorded Open-Science tokens. The domain-adaptation setup and results are detailed in Appendix B.4 and Table 8 (p. 58).

AuthorsOpenReview ↗

Evaluation Protocol (2/4)

W1 — Sandbox and resource limits

Reviewer concern: The paper does not detail the sandbox limits, including whether internet access is blocked and what execution timeouts apply.

We have clarified the sandbox restrictions in the revised manuscript. Agents execute task workflows without network access and cannot retrieve or download data online. They can access the supplied inputs and a writable output directory, while independent observations, expert references, and evaluators remain outside the agent sandbox.

We have also documented CPU, memory, and wall-time limits for all 40 tasks. The resource table reports the maximum configured limits for each task across recorded profiles and runs. The execution environment and complete per-task limits are provided in Appendix E and Table 12 (p. 68).

W2 — Potential memorization of source papers

Reviewer concern: I suspect potential memorization of the source papers. This should be discussed in the limitations sections.

We agree that source-paper memorization is a plausible risk. Models may have encountered the source papers during pretraining, and this knowledge could help them complete the tasks.

Our benchmark evaluates whether agents can execute complete workflows to produce scientifically reliable data products. Knowing a paper's method does not automatically ensure correct data processing or implementation. Even in Base, where methods are explicitly provided, 59.2% of outcomes pass scientific-consistency checks, but only 15.0% reach expert-reference quality (Table 3, p. 9). Across six matched models, method guidance raises scientific consistency by 8.1 percentage points without improving expert-level attainment (Table 4, p. 10).

We have added this limitation to the revised manuscript. The discussion acknowledges possible pretraining exposure and notes that periodically adding tasks from newly published papers could reduce this risk. See Appendix A, “Limitations and Future Work” (p. 15).

W4 — Consistency of the cumulative scoring ladder

Reviewer concern: Several appendix tasks use top-down branching, contradicting the cumulative scoring ladder in Section 3.3.

We acknowledge the inconsistency and have corrected the evaluator control flow and clarified the common bottom-up rule for all 40 tasks. Evaluators check the requirements for 0.2, 0.4, 0.6, 0.8, and 1.0 in ascending order, stop at the first failed gate, and award the highest consecutively passed level. The 0.6 level requires both agreement with the expert product C within the specified tolerance and non-inferiority on independent observations B. Higher levels retain these requirements and add independent-validation improvement and robustness checks.

All 40 task-specific scoring introductions now state ascending evaluation and first-failure stopping, and the scoring rows describe cumulative passing requirements. For example, the Biosphere/EFT 0.8 level retains the 0.6 thresholds of S ≥ 0.92 and F ≥ 0.95, then requires an improvement of at least 0.25 month in independent circular-month error. The 1.0 level inherits all 0.8 requirements and requires this improvement on both site-disjoint and site–year-disjoint partitions. The former S ≥ 0.90 branch therefore cannot bypass the 0.6 requirements.

The cumulative definition is given in Section 3.3 (pp. 6–7) and Table 2 (p. 6). Ascending evaluation, first-failure stopping, and inheritance are now stated explicitly in Appendix A (p. 15), with the Biosphere/EFT example in Appendix A.16 (p. 31).

AuthorsOpenReview ↗

Failure Analysis (3/4)

W3 — Why outcomes stop below expert quality, and what the successful cases show

Reviewer concern: There is no breakdown of why most passing runs stall just below expert quality, no analysis of the three runs that actually beat the expert baseline.

We have added analyses of both the barriers to expert-reference quality and the outcomes that surpass the reference. The updated analysis identifies the delivery failure or first unmet scientific requirement for all 1,000 Base outcomes.

(a) Why outcomes stop at 0.4

The main bottlenecks are expert-reference fidelity and accuracy on independent observations. Among 592 scientifically consistent outcomes, 442 (74.7%) remain at 0.4. We have completed record-level diagnosis for all 442 outcomes, with the following breakdown.

Diagnosed barrier to reaching 0.6Outcomes
Expert-reference fidelity (C)160
Non-inferiority on independent observations (B)147
Combined C/B failures or a failed joint C/B gate135

Passing basic scientific checks therefore does not ensure agreement with the expert product or comparable accuracy on independent observations. Both requirements must be met to reach expert-reference quality. For example, the ChinaAI-FSC run passes independent-validation non-inferiority but exceeds the spatial agreement tolerance to the reconstructed reference, so it remains at 0.4 (Appendix D.2, p. 62).

(b) What the beyond-reference outcomes show

We have reassessed the original high-scoring runs and analyzed the expanded results under the revised scoring rules. Of the original three beyond-reference outcomes, TWSTORE retains 0.8, while the two HUST-GRACE outcomes now score 0.6 because their C20 improvements do not pass the 12-month moving-block bootstrap test. The expanded Base evaluation contains seven outcomes at 0.8 across TWSTORE and ID72, with none at 1.0.

TaskOutcomes at 0.8Improvement over the expert referenceWhy 1.0 is not reached
Water-storage reconstruction (TWSTORE)6A GLM-5.3/Claude Code example reduces independent 1993–2016 centered RMSE from 9.82 to 6.85 mm.Each outcome fails the stronger improvement requirements in at least one early or late validation period.
Forest-height mapping (ID72)1GPT-6-Astra/Codex reduces independent airborne-lidar MAE by 12.2% at Howland and 14.7% at Harvard.Non-inferiority fails for the subgroup of canopies at least 30 m tall.

Most expert-level outcomes still lack sufficient evidence of improvement beyond the reference. Of the 143 outcomes that stop at 0.6, 122 fail the independent-improvement requirement and 21 fail robustness or statistical-evidence requirements (Appendix C.1 and Table 10, p. 59).

The workflow analysis suggests gains from fitting and data fusion within established methods. ID72 combines terrain-corrected radar features with GEDI height samples and selects gradient-boosted regression using spatially blocked validation. A separate skill-augmented TWSTORE run uses ridge regression and validation-weighted fusion of climate and geodetic components. These examples suggest useful implementation choices, while the remaining temporal and tall-canopy failures show that overall gains do not yet establish robust improvement across periods or subgroups.

The failure analysis is summarized in Section 4.3 (p. 10) and detailed in Appendix C.1 and Table 10 (p. 59). The beyond-reference results and workflow examples are discussed in Appendix C.3 (p. 60).

Reviewer_qHEeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EarthDataBench, a benchmark of 40 executable Earth-science data-production tasks adapted from published workflows. Agents produce scientific data products and supporting artifacts. Task-specific, cumulative evaluators assess delivery, scientific checks, agreement with expert-verified reference products, and performance on withheld validation data. Across ten model–harness configurations, 52.0% of outcomes reach the paper’s scientific-consistency level, 19.3% reach expert-reference quality, and 0.8% reach its beyond-reference level. The paper also studies method guidance and failure categories.

Critical strengths
  1. A useful artifact-level benchmark. The tasks require complete data products and supporting materials across seven Earth-science topics. This makes the benchmark relevant to scientific workflows that are difficult to assess through question answering alone.
  2. A carefully motivated evaluation design. The authors reconstruct and expert-check reference products, calibrate task-specific evaluators with valid and deliberately flawed outputs, and state that thresholds are fixed before model comparison. Assessing agent and reference products on the same withheld validation data is valuable. The large difference between the S2 and expert-reference attainment rates identifies a substantial capability gap under this protocol.
Critical weaknesses
  1. The dataset contains only 40 distinct tasks across seven scientific topics. This limits how broadly the results, especially model comparisons, can be generalized.
  2. The scoring scheme requires agreement with the expert reference for higher scores. It may therefore underrate scientifically valid solutions that take a different approach.
Questions for authors
  1. How representative are the 40 tasks of Earth-science data-production work more broadly?
  2. How does the benchmark avoid penalizing valid solutions that differ from the expert reference?
4 comments
AuthorsOpenReview ↗

Task Coverage (1/2)

We thank the reviewer for recognizing the value of artifact-level evaluation and independent validation in EarthDataBench.

We have standardized the cumulative scoring rules across all 40 tasks, updated the results, and expanded the Base evaluation to 1,000 outcomes across 17 model–harness configurations and 11 LLMs, including repeated evaluations. We have also added analyses of the barriers to expert-reference quality and concrete examples of improvements on independent validation.

W1 / Q1 — Representativeness and scope of the 40 tasks

Reviewer concern: How representative are the 40 tasks of Earth-science data production, and how far can the model comparisons be generalized?

We agree that 40 tasks cannot exhaustively represent Earth-science data production. Our publication-derived workflows provide broad coverage across four complementary dimensions, contextualized by established classifications (Figure 1, p. 4, and Appendix A, pp. 15–55):

  • Scientific topics span major Earth-system components. Following NASA GCMD Science Keywords, the tasks cover Atmosphere, Biosphere, Cryosphere, Land Surface, Terrestrial Hydrosphere, Oceans, and Solid Earth. Topics such as Paleoclimate and Sun–Earth Interactions remain outside the suite.
  • Input data span the major scientific data-source families. Our 29 subtypes across six families—satellite observations, in-situ observations, laboratory data, model outputs, reanalysis, and derived products—represent observational, experimental, simulated, and derived inputs in common source-based classifications. Twenty-six tasks combine multiple families.
  • Processing mechanisms cover major scientific data-production operations. Ten mechanisms encompass quality control, multi-source fusion, aggregation, statistical estimation, spatial interpolation, physical retrieval, gap filling, uncertainty quantification, time-series analysis, and classification. Together, they address spatial, temporal, and thematic processing functions described in the ISO 19119/INSPIRE service taxonomy.
  • Product types span major scientific data representations. Five families and 17 subtypes include continuous fields, point products, categorical maps, volumetric fields, and event catalogs. The field and point-based products cover both gridded and non-gridded representations, and continuous and categorical values, consistent with OGC coverage models.

Appendix D (pp. 61–67) illustrates one run from each topic, pairing agent reports and scientific products with evaluator evidence and expert comparisons.

This breadth across major categories does not imply exhaustive coverage of scientific topics, finer-grained data and product subtypes, specific methods, or workflow combinations. Model comparisons reflect performance on this task suite, with broader generalization requiring a larger and more diverse task set. We discuss these coverage limitations in Appendix A ("Limitations and Future Work", p. 15).

AuthorsOpenReview ↗

Reference Tolerances (2/2)

W2 / Q2 — Valid solutions that differ from the expert reference

Reviewer concern: How does the benchmark avoid penalizing scientifically valid solutions that differ from the expert reference?

Task-specific tolerances accommodate numerical differences from the expert reference. To reach 0.6, products must pass scientific-consistency checks, meet these tolerances, and be non-inferior to the expert product on independent observations (Section 3.3, pp. 6–7, and Table 2, p. 6). For example, TWSTORE requires a global time-series correlation ≥0.70 and centered RMSE ≤20 mm against the reference, together with spatial and regional checks. Its independent-validation centered RMSE may be up to 1.05 times the expert value (Appendix A.29, p. 44).

Reference agreement and independent validation provide complementary evidence of product quality. Independent observations may cover only some locations, periods, or variables, so improvements on that subset alone may not establish the quality of the full product. Reference agreement provides additional evidence about the product's broader spatial and temporal structure. Scientifically valid products outside the specified reference tolerances can still be capped at 0.4. The ChinaAI-FSC example has lower held-out RMSE than the reconstructed reference but exceeds the spatial agreement tolerance, so it remains at 0.4 (Appendix D.2, p. 62).

Higher scores reward improvements over the expert product on independent observations. Scores of 0.8 and 1.0 require the preceding criteria plus predefined improvements and task-specific robustness conditions. In ID72 forest-height mapping, GPT-6-Astra/Codex reduced mean absolute error against independent airborne lidar by 12.2% at Howland and 14.7% at Harvard relative to the expert product, earning 0.8 (Appendix C.3, p. 60).

AuthorsOpenReview ↗

Revised Manuscript and Key Updates

We have revised the main text and appendices and submitted an updated manuscript. The main revisions include:

  • Expanded evaluation to 1,000 Base outcomes, including repeated runs and additional baselines.
  • Clarified task coverage and limitations.
  • Added scoring explanations and representative task examples.

Regarding your specific concerns:

  • Representativeness is explained across scientific topics, inputs, processing mechanisms, and products, with explicit limits on generalization beyond the 40 tasks.
  • We clarify how reference tolerances accommodate numerical differences and how independent validation assesses accuracy and improvement, while acknowledging the limitations of these criteria.

We would appreciate your review of the revised manuscript and our responses.

Dear Reviewer qHEe,

Thank you again for your positive assessment of our work and your constructive suggestions. As the rebuttal period draws to a close, we would appreciate your feedback on the revisions addressing your remaining concerns:

  • W1/Q1 (Representativeness): Clarified coverage across scientific topics, inputs, processing mechanisms, and products, added representative examples, and explicitly acknowledged the limits of generalizing beyond the 40 tasks.
  • W2/Q2 (Alternative solutions): Explained how reference tolerances accommodate numerical differences and independent validation assesses accuracy and improvement, supported by concrete examples and a discussion of the criteria’s limitations.

We hope these revisions strengthen your confidence in the paper. If they resolve your remaining concerns, we would be grateful if you could consider strengthening your recommendation. Thank you again for your support and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.