- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes "Extrema", an LLM-powered system for research of extreme weather. Extrema has three main components: "Extremaworld" provides a set of meteorological datasets and scientific tools, and "ExtremaAgent" combines planning, experimentation, reporting and scientific reviews, while "ExtremaBench" evaluates the rediscovery of scientific findings through several tasks from 19 studies, scoring scientific conclusions and figures against expert-based references.
Extrema claims to outperforms the strongest evaluated benchmark and against codex by a significant manager. The ablations suggest there are significant benefits from using scientific tools through agentic AI and a comparatively smaller benefit from reviewing. The supplementary material is extensive and provides data resources, tool inventories, additional resulsts, details on tasks, reports and evaluations.
Critical strengths
- This paper addresses a concrete and important problem in scientific-agent failure modes, which is that the generated executable code can implement incorrect physical methods. The provided diagnostic errors and the significant ablation study provide evidence for the need of additional tooling and reviewsing, and the usefulness of reusable scientific implementaitons.
- The paper uses extreme weather as the main use case, which is a key scientific problem in earth sytem modelling, and I believe it to be a well chosen benchmark for this sort of tooling, as it must deal with out of distribution, tail-effect phenomena.
- The benchmark studies both scientific conclusions and visual evidence (figures), which I think is particularly interesting. The expert reproduction of findings, comparisons and the detailed supplementary examples can provide useful foundations for evaluating research agents for science.
Critical weaknesses
- The evaluation itself is very limited in its independent validation. The supplementary is extensive and interesting and makes individual judgements easier to inspect but does not fully assess agreement with blind review experts or robusness across judges. This is important as the error rate claims are highly dependent on the evaluator itself. Withholding papers are execution time is not sufficient evidence for establishing independent rediscovery of scientific findings.
- Comparisons somewhat confound the contributions of the agent design with those of the environment itself. It's unclear whether the baselines have access to ExtremaWorld data and tools; and the paper says the tools also works with Codex or claude code but the baselines seem to have been run without them. The gains seem to come from the toolset, not the agent architecture (see Table 5). This requires further explanation. It would also be interesting to compare the performance and impact of using Claude Code or Codex, and have these versioned as they are constantly evolving tools.
- The same model (gpt 5.6 sol) seems to have been used as both the judge and the backbone of extremaagent and codex. This is quite problematic as these models tend to show self-preference bias (see Self-Preference Bias in LLM-as-a-Judge, 2024).
- There are no variance estimates or seeded runs, making it hard to assess robustness of the method.
- The paper motivates its system as it can democratise research and reduce inequality. This is never tested, and I would argue that having LLM-driven research agents that run for 2 days for each task is not exactly democratising science as these tools can be quite expensive and dependent on the prices of private companies.
Questions for authors
- Do the baselines have access to the ExtremaWorld tools and data?
- What are the agreement patterns between LLM judges and human expert; and with other LLMs (ideally non-gpt)?
- What would be the impact of using open-weights or locally-running models?
- Will the benchmark tasks, reference materials, etc, be released publicly? The reproducibility statement mentions mostly evaluation scripts, which I think are insufficient for reproducibility.