- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents EarthDataBench: dozens of long-horizon tasks, curated from papers in an Earth-science data journal, that require turning raw observational data into research-quality data products across several Earth-science domains. Evaluation uses a five-level cumulative score, from "usable deliverables" through "scientifically consistent" and "at expert level" to "surpassing the expert product on independent observations." Evaluating several LLMs under two agent frameworks, the authors find that about half of the outcomes are scientifically consistent, about one fifth reach expert level, almost none surpass the expert, and format problems are the leading source of failure.
Critical strengths
- Valuable topic, clearly distinct from existing Earth-science benchmarks. Requiring agents to deliver complete data products (with code and reports) from raw data is much closer to real scientific work than the common QA-style evaluations.
- Expert involvement and calibration awareness in construction. Reference products are manually verified, and the task cards are thoroughly disclosed, making it possible to audit the evaluators themselves.
Critical weaknesses
-
The unified scoring protocol promised in the main text is not implemented uniformly across the tasks' actual rules, so the headline numbers are not on a common footing. The main text says the five levels are strictly cumulative and that "expert level" requires both closeness to the expert product and being no worse than the expert on independent observations. But in the appendix task cards: some scores are not cumulative (in the Biosphere task, the 0.8 level requires similarity S ≥ 0.90, lower than the 0.6 level's S ≥ 0.92, so a product can earn 0.8 without qualifying for 0.6); only about three tasks actually check both conditions; the "expert reference" is sometimes the published product, sometimes an AI-assisted reconstruction, and sometimes just a baseline model; and four tasks list thresholds as "not provided." As a result, the abstract's "52% pass scientific checks, 19.3% reach expert level" does not mean the same thing across tasks and needs to be recomputed under a unified standard.
-
The "beyond expert" level is barely measurable in the current data, and single-run results plus unspecified run conditions add uncertainty. Only 3 of 400 outcomes reach "beyond expert," and the top level is 0; some tasks are hard to surpass by design (e.g., tasks that require copying native values directly). All results come from a single run (each task is worth 2.5 percentage points, so the 7.5-point framework difference amounts to 3 tasks, and inter-model differences are of the same magnitude); the paper does not state whether the execution environment has network access (the tasks come from an open-access journal, so agents could potentially fetch the published products directly), nor the hardware or time budgets.
Questions for authors
Clarifications needed
- Please provide the level-by-level decision rules for all 40 tasks: for how many tasks does "expert level" check both consistency with the expert product and independent observations? For how many is the scoring not strictly cumulative? If recomputed under a unified standard, how would the abstract's 52.0% / 19.3% change?
- Which tasks do the 3 "beyond expert" results come from? For each task, have you verified that "beyond expert" is achievable in principle (e.g., with an oracle or a known better method)?
- Does the execution environment have network access? Have you checked whether agents directly download the published products?
Suggestions
- Add comparisons with general science / data-science agent benchmarks such as ScienceAgentBench, MLE-bench, and DSBench.
- Report format-related zero scores separately to distinguish "delivery problems" from "scientific capability."
- Provide at least a few complete task packages (with evaluators and thresholds) before release to support community reproduction.
4 comments
Results and Reliability (2/2)
W2 / Q2 — Beyond-reference outcomes
Concern summary: Beyond-reference outcomes and the attainability of higher scores.
We address both score-level attainability and the observed beyond-reference outcomes. The outcome analysis has been added to the revised manuscript (Appendix C.3, p. 60).
Beyond-reference attainability was verified during task construction. We used expert-reference products and constructed oracle test cases to calibrate each task evaluator across score levels. The oracle tests confirmed that the cumulative requirements for 0.8 and 1.0 could be satisfied (Section 3.4, p. 7).
(a) Original beyond-reference outcomes
The original three beyond-reference outcomes came from two tasks. Two were HUST-GRACE reconstructions by GPT-6-Astra and GPT-6-Luna with Codex, and one was TWSTORE with GLM-5.3-Flash/Claude Code. Under the revised rules, both HUST-GRACE outcomes score 0.6 because their C20 improvements fail the 12-month moving-block bootstrap test required for 0.8 (Appendix A.39, p. 54).
The TWSTORE outcome retains 0.8, but does not reach 1.0 because the required improvement in the 2011–2016 period is not achieved. The temporal validation requirements are specified in Appendix A.29 (p. 44).
(b) Additional evidence from new experiments
The expanded Base cohort contains seven 0.8 outcomes: six on TWSTORE and one on ID72, with none at 1.0 (Appendix C.3, p. 60). For TWSTORE, the GLM-5.3/Claude Code example reported in the manuscript reduces independent 1993–2016 centered RMSE from 9.82 to 6.85 mm. All six TWSTORE outcomes fail the stronger improvement requirements in at least one early or late validation period.
For ID72 forest-height mapping, GPT-6-Astra/Codex combined terrain-corrected radar features with GEDI height samples and selected a gradient-boosted regressor using spatially blocked validation. Independent airborne-lidar mean absolute error decreases by 12.2% at Howland and 14.7% at Harvard relative to the expert reference. Tall-canopy errors fail the non-inferiority requirement for 1.0, so the result remains at 0.8 (Appendix C.3, “Forest-height mapping,” p. 60).
Most expert-level outcomes still lack sufficient evidence of improvement beyond the reference. Among the 143 outcomes that stop at 0.6, 122 fail the independent-improvement requirement and 21 fail robustness or statistical-evidence requirements (Appendix C.1 and Table 10, p. 59). Improvement headroom remains task-dependent. In the exact-assembly example, reproducing the expert product yields 0.6 but no independent-validation gain (Appendix D.7, p. 67).
W2 — Reliability of repeated evaluations and model comparisons
Concern summary: Reliability of single-run results and small differences between configurations.
To strengthen reliability, we evaluated four representative model–harness configurations three times each on the same 40 Base tasks: GPT-6-Astra, GLM-5.3, and Kimi K3.1 Pro with Codex, and GLM-5.3 with Claude Code. Time and computational costs prevented us from repeating every configuration. We now report individual results, means, and sample standard deviations (Appendix B.3, pp. 57–58, Table 7, p. 58). Mean-score standard deviations range from 0.29 to 1.15 points on the 0–100 scale, indicating stable aggregate performance for these configurations. Closely performing configurations can nevertheless change order between runs, so small differences do not establish definitive rankings.
Across all 12 evaluations, scientific-consistency attainment exceeds expert-reference quality by 42.5–57.5 percentage points. This persistent gap supports our central claim that reliable scientific execution remains a key bottleneck even when methods are prescribed (Section 4.3, p. 10).
W2 / Q3 — Execution conditions and potential product downloading
Concern summary: Execution settings and the risk of retrieving published products.
We have added the execution environment and resource limits for all 40 tasks (Appendix E, Table 12, p. 68). Agents run in a sandbox without network access, so they cannot retrieve or download published products online. Public inputs and writable outputs are accessible, while independent observations, expert references, and evaluators remain outside the sandbox. CPU, memory, and time limits are configured separately for each task.
Revised Manuscript and Key Updates
We have revised the main text and appendices and submitted an updated manuscript. The main revisions include:
- Expanded evaluation to 1,000 Base outcomes, including repeated runs.
- Standardized scoring rules and clarified reference sources and thresholds.
- Added benchmark comparisons, failure analyses, and reproducibility details.
Regarding your specific concerns:
- All 40 tasks now follow the cumulative scoring rule, and the Biosphere inconsistency has been corrected.
- Repeated evaluations and revised beyond-reference analyses clarify result stability and barriers to higher scores.
- Delivery and scientific failures are distinguished, with offline execution conditions documented and representative task packages provided.
We would appreciate your review of the revised manuscript and our responses.
Request for Feedback and Reconsideration
Dear Reviewer XNjp,
Thank you again for your time and thoughtful feedback. As the rebuttal period draws to a close, we would appreciate your review of our revised manuscript and responses. We briefly summarize the updates below:
- W1/Q1 (Scoring): Unified cumulative scoring across all 40 tasks, clarified references and thresholds, and updated results to 1,000 outcomes.
- W2/Q2 (Higher scores): Clarified evaluator calibration and analyzed beyond-reference outcomes and barriers to higher scores.
- W2 (Reliability): Added three runs for four configurations, supporting the main conclusion while qualifying close rankings.
- W2/Q3 (Execution): Documented offline execution, evaluation-data isolation, and per-task resource limits.
- S1 (Comparisons): Added comparisons with ScienceAgentBench, MLE-bench, and DSBench.
- S2 (Failures): Separated 140 delivery failures from 41 scientific failures.
- S3 (Reproducibility): Provided seven representative task packages, including evaluators and thresholds.
We would be grateful to hear whether any concerns remain. If these revisions address your concerns, we would appreciate it if you could reconsider your assessment and score. Thank you again for your time and consideration.
Scoring and Reproducibility (1/2)
We thank the reviewer for recognizing the value of artifact-level evaluation and expert involvement in EarthDataBench.
We have revised the scoring rules and reference documentation, updated the results, and expanded the Base evaluation to 1,000 outcomes. We also added repeated evaluations, separated delivery failures from scientific failures, and documented the offline sandbox and per-task resource limits.
W1 / Q1 — Consistency of the scoring protocol and revised results
We completed a systematic review and update of the scoring rules, evaluators, and task briefs for all 40 tasks.
(a) Cumulative scoring
All 40/40 tasks now use strictly cumulative scoring, and 40/40 require both agreement with the expert product C and non-inferiority on independent observations B at 0.6. Higher levels inherit all preceding requirements under the common framework (Section 3.3, pp. 6–7). The Biosphere inconsistency is corrected: 0.8 now retains the 0.6 requirements of S ≥ 0.92 and F ≥ 0.95 (Appendix A.16, p. 31). The revised task-specific rules are provided for all 40 tasks (Appendix A, pp. 15–55).
(b) Reference products and thresholds
For 39 tasks, references are author-released products or their deterministic derivatives. ChinaAI-FSC uses an explicitly labelled AI-assisted reconstruction of the author's UNet method because the original weights and test predictions are unavailable (Appendix A.10, p. 25). The task briefs clarify these sources and replace the former “not provided” entries with task-specific scoring rules and threshold descriptions (Appendix A, pp. 16–55). Remaining private numerical thresholds are explicitly identified.
(c) Revised results
The revised evaluation preserves the main finding that scientific consistency is substantially more common than expert-reference quality. The original submission and current expanded evaluation are summarized below.
The full Base row reflects both revised scoring and additional experiments. It covers 1,000 outcomes across 25 result sets, 17 model–harness configurations, and 11 LLMs (Section 4.2, p. 8 and Table 3, p. 9). These results retain the central conclusion: reliable execution remains a bottleneck even when scientific methods are prescribed.
S1 — Comparison with general scientific and data-science benchmarks
We have added discussion and comparisons of general scientific and data-science benchmarks, including ScienceAgentBench, MLE-bench, and DSBench, in the main text (Section 2, pp. 2–3 and Table 1, p. 3).
S2 — Separating delivery failures from scientific-quality failures
Delivery failures and scientific invalidity are now diagnosed separately. Of the 181 zero-score outcomes, 140 fail before scientific scoring and 41 fail minimum scientific validity after delivery (Section 4.3, p. 10, and Appendix C.1, Table 9, p. 59). The updated appendix identifies the delivery failure or first unmet scientific requirement for all 1,000 Base outcomes, including barriers at higher score levels (Appendix C.1 and Table 10, p. 59). The LST example illustrates successful delivery with a zero scientific score because predictions extend outside the permitted spatial support (Appendix D.5, p. 65).
S3 — Reproducible task packages
We have provided one complete representative task package from each Earth-science domain, including evaluators and scoring thresholds. The packages are available at the anonymous sample repository.