acceptodds
Withdrawn from ICLR 2027

Extrema: Enabling and Evaluating Agentic Research on Extreme Weather

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding extreme weather is vital for climate risk management, but it demands scarce domain expertise. This barrier deepens global inequalities in climate resilience. AI agents offer a path toward democratizing such research. However, existing agents struggle with long-horizon inquiry, as error-free code cannot guarantee physical validity. We introduce Extrema, a system combining ExtremaWorld—which unifies 20 datasets and 100 validated tools—with ExtremaAgent, which pairs a scientific review loop and a persistent task graph to guard against methodological errors. To evaluate such investigations, we introduce ExtremaBench, which distills 58 tasks from 19 published studies covering five extreme weather categories. Withholding the source study, we require each agent to rediscover the study's findings independently, and grade the resulting conclusions and supporting figures against expert-authored rubrics. Powered by GPT-5.6-sol, Extrema scores 71.58/100, outperforming the strongest baseline by 10.07 points.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Extreme Weather, LLM Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_3P4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "Extrema", an LLM-powered system for research of extreme weather. Extrema has three main components: "Extremaworld" provides a set of meteorological datasets and scientific tools, and "ExtremaAgent" combines planning, experimentation, reporting and scientific reviews, while "ExtremaBench" evaluates the rediscovery of scientific findings through several tasks from 19 studies, scoring scientific conclusions and figures against expert-based references.

Extrema claims to outperforms the strongest evaluated benchmark and against codex by a significant manager. The ablations suggest there are significant benefits from using scientific tools through agentic AI and a comparatively smaller benefit from reviewing. The supplementary material is extensive and provides data resources, tool inventories, additional resulsts, details on tasks, reports and evaluations.

Critical strengths
  • This paper addresses a concrete and important problem in scientific-agent failure modes, which is that the generated executable code can implement incorrect physical methods. The provided diagnostic errors and the significant ablation study provide evidence for the need of additional tooling and reviewsing, and the usefulness of reusable scientific implementaitons.
  • The paper uses extreme weather as the main use case, which is a key scientific problem in earth sytem modelling, and I believe it to be a well chosen benchmark for this sort of tooling, as it must deal with out of distribution, tail-effect phenomena.
  • The benchmark studies both scientific conclusions and visual evidence (figures), which I think is particularly interesting. The expert reproduction of findings, comparisons and the detailed supplementary examples can provide useful foundations for evaluating research agents for science.
Critical weaknesses
  • The evaluation itself is very limited in its independent validation. The supplementary is extensive and interesting and makes individual judgements easier to inspect but does not fully assess agreement with blind review experts or robusness across judges. This is important as the error rate claims are highly dependent on the evaluator itself. Withholding papers are execution time is not sufficient evidence for establishing independent rediscovery of scientific findings.
  • Comparisons somewhat confound the contributions of the agent design with those of the environment itself. It's unclear whether the baselines have access to ExtremaWorld data and tools; and the paper says the tools also works with Codex or claude code but the baselines seem to have been run without them. The gains seem to come from the toolset, not the agent architecture (see Table 5). This requires further explanation. It would also be interesting to compare the performance and impact of using Claude Code or Codex, and have these versioned as they are constantly evolving tools.
  • The same model (gpt 5.6 sol) seems to have been used as both the judge and the backbone of extremaagent and codex. This is quite problematic as these models tend to show self-preference bias (see Self-Preference Bias in LLM-as-a-Judge, 2024).
  • There are no variance estimates or seeded runs, making it hard to assess robustness of the method.
  • The paper motivates its system as it can democratise research and reduce inequality. This is never tested, and I would argue that having LLM-driven research agents that run for 2 days for each task is not exactly democratising science as these tools can be quite expensive and dependent on the prices of private companies.
Questions for authors
  • Do the baselines have access to the ExtremaWorld tools and data?
  • What are the agreement patterns between LLM judges and human expert; and with other LLMs (ideally non-gpt)?
  • What would be the impact of using open-weights or locally-running models?
  • Will the benchmark tasks, reference materials, etc, be released publicly? The reproducibility statement mentions mostly evaluation scripts, which I think are insufficient for reproducibility.
Reviewer_KqgEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper first identifies one major problem of using general purpose agent like codex and claude code in extreme whether research: they are bad at domain-specific analysis when coding from scratch. Therefore, they propose ExtremaWorld, a combination of domain-specific knowledge source, datasets, and a well packaged domain-specific toolbox with 100 tools. They also made ExtremaAgent, a planning -> execution -> review agent that solves problem in ExtremaWorld. They futher introduce ExtremaBench, a PaperBench-like benchmark that test whether the agents can reproduce the results of research papers. There system (ExtremaWorld + ExtremaAgent) shows substantial improvement on ExtremaBench.

Critical strengths
  1. The research topic is very important and meaningful.
  2. If the evaluation is reliable, the improvement on ExtremaBench is substantial compared to the codex baseline.
  3. Table 4 shows interesting failures of frontier agents, motivating this research well.
Critical weaknesses
  1. Only GPT-5.6-sol is used as the judge and no agreement to human expert is measured.
  2. I found no description about the internet access of the agents. If they have internet access, will there be a chance of answer leakage? If they don't, it may restrict codex's & claude code's ability a lot, making the result poorly indicate the real usage of these agents.
  3. Ablating the toolbox leads to the most drastic performance drop, making me concern that the improvement is mainly contributed by the toolbox instead of the design of ExtremaAgent. Can you add an experiment by giving codex/claude code the exact toolbox?
Questions for authors

Appendix A and B are missing from the paper.

Reviewer_uNBjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Extrema combines a scientific computing environment for extreme-weather research with an agent that plans analyses, executes experiments, and writes reports, supported by a persistent task graph and independent scientific review. ExtremaWorld integrates 20 data resources and 100 reusable scientific tools. ExtremaBench evaluates rediscovery of published findings through 58 tasks from 19 papers, with expert-authored references covering both scientific claims and supporting figures. Experiments compare complete agent systems, analyze scientific errors, and ablate tools and review on a task subset.

Critical strengths

The most useful contribution is the concrete analysis of scientific implementation failures, including omitted latitude factors, tracking errors, and missing potential-vorticity terms (Table 4). Connecting these examples with the tool and review ablations offers practical insight into the failures that reusable scientific software can help prevent. The integrated data and tool interfaces also have resource value, while evaluating both conclusions and supporting figures is a useful benchmark design. I appreciate the explicit examples of missed interpretations and the acknowledgment that generated evidence does not guarantee a scientifically complete report or reliable climate-risk assessment.

Critical weaknesses

1. The contribution of the agent design is not sufficiently separated from the curated environment. In Table 5, Codex is evaluated without the provided scientific toolset, although the paper says these tools work with other coding agents. The exact data, tool-catalog, and permission conditions for the main comparison are not documented clearly enough to establish a matched environment. The authors acknowledge comparing complete systems, and resource integration is valuable, but this leaves the incremental contribution of the orchestration unclear. On the ablation subset, the full system scores 79.2, removing review gives 74.2, and removing tools while retaining review gives 57.1, below Codex at 61.7. The subset size and selection are unspecified, and there is no task-graph ablation. These results do not establish which design elements explain the broader gains. A comparison with shared tools and data, plus the subset manifest and paired results, would materially improve this evidence.

The practical accessibility argument is also insufficiently evaluated. Table 3 supports lower adjusted token usage, but includes cached inputs and excludes confirmed redundant empty polling. It does not establish lower monetary cost or total resource demand without wall time, cache-aware cost, and tool-compute accounting. I recognize that the table itself claims token efficiency; broader claims about making research accessible need additional evidence or a narrower scope.

2. The evaluation does not establish sufficiently reliable scientific-quality or generalization conclusions. GPT-5.6-sol judges both claims and figures, but expert task construction does not validate this judge: no independent expert agreement, blinded rescoring, or alternative-judge sensitivity is reported. Sharing the backbone with both ExtremaAgent and matched Codex does not by itself prove bias, but it does not resolve calibration. The 58 tasks come from 19 papers, yet full per-task scores, repeated-run variability, and uncertainty accounting for source-paper clustering are not reported. This limits confidence in the stability and breadth of the reported improvements.

The rediscovery protocol also needs an auditable isolation description. The paper explicitly withholds target papers and references, but the tool inventory includes web search and retrieval; run-specific restrictions and target-source access audits are not reported. This is an unresolved control, not evidence that leakage occurred. Finally, tasks are filtered for feasible reproduction and figure-supported findings. That is reasonable for constructing a benchmark, but coverage across regions, events, and data conditions is insufficiently characterized to support broader research-capability claims. The stated limitations are appropriate; they do not replace validation of those claims. Taken together, the attribution and evaluation gaps prevent me from supporting acceptance.

Questions for authors
  1. Appendix D states that criticism of task-prescribed parameters was removed from the Codex atmospheric-river case explanation without recomputing scores. How was it established that the invalid criticism did not affect the recorded score? Could the case be rescored with the corrected reasoning and the remaining runs audited for the same issue?

  2. How do the 100 validated scientific tools map to the 114 inventory entries? Please provide validation fixtures, numerical tolerances, coverage, and the atmospheric-river detector's deviations from the reference method. I recognize that the inventory includes workflow utilities, so the different counts alone do not establish an inconsistency.

  3. Can you provide an anonymized reproducibility package with all 58 tasks and rubrics, exact agent prompts and configurations, model and command-line interface versions, and per-task logs and scores? These are clarification and validation requests for the reported results, rather than requests for a new application domain.

Reviewer_Jk9cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents Extrema, an agentic framework for extreme weather. It consists of : (i) ExtremaWorld, a collection of 20 datasets and 100 tools; and (ii) ExremeAgent, which pairs a scientific review loop and a persistent task graph to guard against methodological errors.

Critical strengths

None

Critical weaknesses

I do not believe this is an original. It is similar to Zephyrus (Varambally et al., 2026), an agentic framework for weather- and climate-related question-answering tasks. Even though the authors claim their proposed framework has access to more tools or sources, Zephyrus is extensible by nature and can therefore incorporate more tools as needed. Furthermore, the experimental setup appears to be very weak. The authors unfairly compare their method to general-purpose agents, such as Codex and Claude, while omitting weather- or climate-based agentic methods, such as Zephyrus. A fair comparison should involve methods designed to tackle the same or similar tasks.

Questions for authors

Based on concerns raised in the Critical Weaknesses section, this paper does not present anything new. My main question, therefore, is: What is the difference between Extrema and Zephyrus when augmented with the tools mentioned in the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.