- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EarthLitBench, a benchmark with 822 instances built from 112 Earth science review articles and their cited literature. It covers ten tasks, including literature selection, study comparison, mechanism reasoning, and research-gap identification. The proposed RERA evaluation scores source selection, evidence fidelity, reasoning fidelity, and answer fidelity. The experiments compare nine models, with four also evaluated using a shared retrieval pipeline.
Critical strengths
- The tasks capture useful parts of scientific work. Comparing studies, explaining conflicting findings, and deciding whether a result transfers to another setting are meaningful challenges.
- The task boundaries are carefully explained. Appendix A distinguishes finding a relevant study from deciding whether studies are comparable, and distinguishes describing a mechanism from establishing that it transfers to a new setting.
- The evaluation allows valid alternative answers. The protocol does not require models to reproduce the hidden review’s wording or use exactly the reference bibliography. It also allows justified conclusions such as insufficient evidence or unresolved disagreement.
- The paired retrieval comparison is useful. Reusing the same retrieved material across models helps control differences in evidence access. The authors also acknowledge that limited retrieval gains do not, by themselves, establish a limitation in intrinsic reasoning ability.
Critical weaknesses
- The benchmark and judge still need independent validation. This is my main concern. Section 4.2 explicitly says that the reliability results “remain to be reported.” For a benchmark whose main results depend on LLM-assigned scores, these results are essential. Qwen3.8-27B is used for both benchmark construction and judging, and it is also one of the evaluated models. This does not prove that the evaluation is biased, but it makes independent checks particularly important. The generator and judge could share the same mistaken interpretation of a source or the same preference for a particular answer structure. The construction analysis in Section 5.1 reports types of rejected graph records from ten surveys. That shows what the screening process catches, but it does not tell me how many errors remain in the accepted questions, reference answers, or rubrics. I would like to see an expert audit of accepted instances and model responses, with agreement reported separately for the four scoring dimensions.
- The reported scores do not clearly show where models are failing. RERA separates four dimensions, but Table 3 reports only their average. This makes the main reasoning-related conclusions difficult to assess. A low score on a mechanism task could come from missing citations, incorrect evidence, an invalid inference, or an incomplete answer. These are different failures, but is important especially for the closed-book setting. Moreover, I appreciate the authors’ caution about incomplete retrieval. Still, without retrieval-coverage measurements, the small gains on T06–T08 remain difficult to interpret. The pipeline may simply fail to retrieve the evidence needed for those tasks. Please report the four dimensions separately and add an experiment where models receive the verified evidence needed to answer each question.
- There is not enough concrete information to inspect the benchmark. The paper provides detailed task descriptions, but I could not find a complete benchmark example containing the public question, supplied materials, reference answer, supporting passages, and scoring checklist. Such examples are important here: they would let readers check whether a question actually requires cross-study reasoning and whether the hidden rubric asks for anything beyond the public prompt. Several basic details are also missing, including the number of instances per task, the size of the authorized reference corpus, and the development/evaluation split. The paper mentions separate text and visual tracks but does not report their sample sizes or clearly explain how they contribute to Table 3. The retrieval pipeline is described as fixed, but its retriever, chunk size, number of retrieved passages, and context budget are not specified. Please also clarify whether the publicly releasable benchmark will contain all 822 evaluated instances, given the access restrictions discussed in Section 7.
- One of the main experimental conclusions is contradicted by Table 3. Section 5.2 says that proprietary models have a larger advantage on reasoning-intensive tasks than on question decomposition and literature selection. Using the four open-weight and five proprietary LLM-only models listed in Table 3, I obtain the following unweighted mean differences:- T01–T02: approximately 19.1 points. - T06–T08: approximately 12.6 points. The gap is therefore smaller on the reasoning tasks under this calculation. If the authors used a different grouping or weighting, it needs to be explained. The same paragraph still contains the placeholders “[A],” “[B],” and “[C].” Together with the missing reliability results, this makes the experimental analysis appear unfinished.
Questions for authors
NA