acceptodds
Withdrawn from ICLR 2027

EarthLitBench: Benchmarking Literature-Grounded Scientific Inquiry in Earth Science

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scientific inquiry in Earth system science demands synthesizing fragmented, cross-study empirical findings across diverse environmental spheres and temporal-spatial scales. Evaluating LLM support for this work therefore calls for assessing evidence selection and scientific justification together. However, existing benchmarks predominantly focus on single-document factoid QA, leaving a critical blind spot in evaluating multi-hop reasoning and evidence synthesis across extensive literature. To bridge this gap, We introduce EarthLitBench, a benchmark of 822 instances across ten tasks organized into four capability groups: problem formulation and literature discovery, study and evidence understanding, scientific reasoning, and knowledge synthesis. Grounding our benchmark in authoritative scientific reviews, we construct traceable knowledge graphs that capture complex scientific relationships, thereby guiding rigorous question generation and evidence verification. To ensure fair evaluation, source reviews, construction graphs, and reference solutions are strictly withheld from candidate models, while evidence retrieval is constrained within an authorized reference corpus. Furthermore, we establish the RERA evaluation protocol, which couples automated source verification with reference-guided LLM judges across four complementary dimensions: Retrieval, Evidence fidelity, Reasoning fidelity, and Answer fidelity. Experiments evaluate nine LLM-only models and compare four of them with fixed retrieval. Models perform relatively well on problem decomposition but have greater difficulty with scientific reasoning, even though explicit references are given. These findings highlight a gap between accessing scientific literature and conducting scientific inquiry, motivating stronger integration of evidence and scientific interpretation.

Primary area:
datasets and benchmarks
Keywords:
AI for science, Benchmark, Earth Science

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GPGVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EarthLitBench, a benchmark with 822 instances built from 112 Earth science review articles and their cited literature. It covers ten tasks, including literature selection, study comparison, mechanism reasoning, and research-gap identification. The proposed RERA evaluation scores source selection, evidence fidelity, reasoning fidelity, and answer fidelity. The experiments compare nine models, with four also evaluated using a shared retrieval pipeline.

Critical strengths
  1. The tasks capture useful parts of scientific work. Comparing studies, explaining conflicting findings, and deciding whether a result transfers to another setting are meaningful challenges.
  2. The task boundaries are carefully explained. Appendix A distinguishes finding a relevant study from deciding whether studies are comparable, and distinguishes describing a mechanism from establishing that it transfers to a new setting.
  3. The evaluation allows valid alternative answers. The protocol does not require models to reproduce the hidden review’s wording or use exactly the reference bibliography. It also allows justified conclusions such as insufficient evidence or unresolved disagreement.
  4. The paired retrieval comparison is useful. Reusing the same retrieved material across models helps control differences in evidence access. The authors also acknowledge that limited retrieval gains do not, by themselves, establish a limitation in intrinsic reasoning ability.
Critical weaknesses
  1. The benchmark and judge still need independent validation. This is my main concern. Section 4.2 explicitly says that the reliability results “remain to be reported.” For a benchmark whose main results depend on LLM-assigned scores, these results are essential. Qwen3.8-27B is used for both benchmark construction and judging, and it is also one of the evaluated models. This does not prove that the evaluation is biased, but it makes independent checks particularly important. The generator and judge could share the same mistaken interpretation of a source or the same preference for a particular answer structure. The construction analysis in Section 5.1 reports types of rejected graph records from ten surveys. That shows what the screening process catches, but it does not tell me how many errors remain in the accepted questions, reference answers, or rubrics. I would like to see an expert audit of accepted instances and model responses, with agreement reported separately for the four scoring dimensions.
  2. The reported scores do not clearly show where models are failing. RERA separates four dimensions, but Table 3 reports only their average. This makes the main reasoning-related conclusions difficult to assess. A low score on a mechanism task could come from missing citations, incorrect evidence, an invalid inference, or an incomplete answer. These are different failures, but is important especially for the closed-book setting. Moreover, I appreciate the authors’ caution about incomplete retrieval. Still, without retrieval-coverage measurements, the small gains on T06–T08 remain difficult to interpret. The pipeline may simply fail to retrieve the evidence needed for those tasks. Please report the four dimensions separately and add an experiment where models receive the verified evidence needed to answer each question.
  3. There is not enough concrete information to inspect the benchmark. The paper provides detailed task descriptions, but I could not find a complete benchmark example containing the public question, supplied materials, reference answer, supporting passages, and scoring checklist. Such examples are important here: they would let readers check whether a question actually requires cross-study reasoning and whether the hidden rubric asks for anything beyond the public prompt. Several basic details are also missing, including the number of instances per task, the size of the authorized reference corpus, and the development/evaluation split. The paper mentions separate text and visual tracks but does not report their sample sizes or clearly explain how they contribute to Table 3. The retrieval pipeline is described as fixed, but its retriever, chunk size, number of retrieved passages, and context budget are not specified. Please also clarify whether the publicly releasable benchmark will contain all 822 evaluated instances, given the access restrictions discussed in Section 7.
  4. One of the main experimental conclusions is contradicted by Table 3. Section 5.2 says that proprietary models have a larger advantage on reasoning-intensive tasks than on question decomposition and literature selection. Using the four open-weight and five proprietary LLM-only models listed in Table 3, I obtain the following unweighted mean differences:- T01–T02: approximately 19.1 points. - T06–T08: approximately 12.6 points. The gap is therefore smaller on the reasoning tasks under this calculation. If the authors used a different grouping or weighting, it needs to be explained. The same paragraph still contains the placeholders “[A],” “[B],” and “[C].” Together with the missing reliability results, this makes the experimental analysis appear unfinished.
Questions for authors

NA

Reviewer_KF9UOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EarthLitBench for evaluating literature-grounded scientific inquiry in Earth system science. The benchmark spans multiple tasks grouped roughly into problem formulation and literature discovery, evidence understanding, scientific reasoning, and knowledge synthesis. The authors build traceable knowledge resources from reviews, then generate scored evaluation instances under a unified protocol; construction materials are withheld and evidence access is restricted at test time. Experiments compare several LLM-only models and fixed-retrieval controls for a subset, reporting relative strengths across task groups and asymmetric retrieval gains.

Critical strengths
  1. Task stratification plus a unified evidence interface has diagnostic value: scoring discovery, understanding, comparative/mechanistic reasoning, and synthesis separately—while requiring source–evidence–argument–output—makes “retrieval helps organization but not scientific reasoning” a discussable bottleneck rather than one coarse scientific-QA score.
  2. Paired fixed-retrieval controls are a persuasive design: same models and a shared retrieval pipeline help separate “more context” from “can the model complete inquiry with it,” supporting the central claim that access to literature alone does not guarantee completed scientific inquiry.
Critical weaknesses
  1. The validity loop for evaluation is incomplete, undermining the paper’s central claim. The main protocol relies heavily on reference solutions and an LLM judge, while human agreement, sensitivity analyses, and some key group contrasts look under-reported or unfinished; if graph construction and item generation share a model family with judging, that risk is also insufficiently quantified. Without these, score rankings and claims such as “reasoning is harder and retrieval helps little” cannot be treated as reliable scientific conclusions.
  2. Arguments for benchmark quality, contamination control, and difficulty differentiation are insufficient. Generation and human verification are thinly disclosed, cross-task difficulty separation looks weak, and contamination controls mainly hide construction materials without adequately addressing memorization of authorized literature or inflation risk on open-ended or softer item types. These issues directly threaten construction validity and fair comparison, and are not resolved by brief caveats.
Questions for authors
  1. Can you provide human-audit scale, agreement with the LLM judge, and sensitivity of the main conclusions? If incomplete, how should the main tables be caveated?
  2. If graph construction and item generation share a model family with judging, have you run cross-judge or cross-reference-solution re-evaluations?
  3. How much of LLM-only high scores may come from pretraining memory rather than the authorized corpus? For softer synthesis-style tasks, is there a stricter re-annotation or sensitivity check?
Reviewer_qpFBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

EarthLitBench evaluates how language models use scientific literature to answer Earth science questions. It contains 822 instances generated from 112 review articles and their cited studies, covering ten tasks such as interpreting results, comparing studies, explaining mechanisms, and assessing whether findings apply under different conditions. Responses are evaluated on source selection, evidence, reasoning, and answer quality. Experiments compare nine models without retrieval and four with a shared retrieval pipeline.

Critical strengths

The problem statement is interesting and potentially useful.

The proposed benchmark includes useful scientific tasks beyond factual question answering, such as explaining why studies disagree and determining whether a finding applies in another setting.

The evaluation seeks to ask models to identify supporting sources and evidence, allowing the scientific basis of an answer to be examined alongside its conclusion.

Critical weaknesses

A real weakness is that the paper does not show a complete question, the materials given to the model, an expected answer, or its scoring. I cannot get a clear sense of what the tasks require or assess their difficulty. Each of the proposed tasks need to accompanied by and end to end expert constructed example. It would also then help to see such an example created by an llm. Then, the paper can give the "misalignment" according to its metrics.

I have concerns about correctness of the benchmark. An LLM generates the questions, reference answers, explanations, and rubrics. Human checks are mentioned, but their scope and results are not reported. Reporting rejected graph records does not establish that the accepted instances are correct.

Some claims are not fully supported by the reported statistics. For example, the paper claims that proprietary models have a larger advantage on reasoning tasks, but Table 3 shows a smaller average gap on T06–T08 (12.6 points) than on T01–T02 (19.1 points).

I believe that the aggregated results obscure where models struggle. By combining source selection, evidence, reasoning, and answer quality into one score, the paper makes it difficult to understand the failures or support its conclusions about reasoning limitations.

The paper omits task sample sizes, retrieval and execution settings, and uncertainty estimates. It also leaves “[A],” “[B],” and “[C]” placeholders in the results and cites Appendices D and F, which are missing. This suggests a rushed submission.

Parts of the paper are not easy to follow. The paper requires further proofreading.

Questions for authors

Please see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.